Repository navigation
logd: don't hand out stale entries after the ring wraps - #9
Conversation
…(B162) openwrt/ubox#9 as a ubox package patch (r2): fixes the occasional logread SIGSEGV (FS#4118 / openwrt#9100) seen in the C-360s' logread pipelines. W-318 reproducer: stock 6 crashes in 150 s, patched 0 in 300 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Tested on real hardware: Arista W-318 (Qualcomm IPQ6018, aarch64, musl), OpenWrt main with ubox 6f78fa4, 128 KiB ring ( FS#4118 reproducer as in your description:
We hit it in the field first, on IPQ8074 APs: twice in nine days, in Tested-by: Christopher Griffin christophergriffin05@gmail.com 🤖 Tested and written up with Claude Code |
1 similar comment
|
Tested on real hardware: Arista W-318 (Qualcomm IPQ6018, aarch64, musl), OpenWrt main with ubox 6f78fa4, 128 KiB ring ( FS#4118 reproducer as in your description:
We hit it in the field first, on IPQ8074 APs: twice in nine days, in Tested-by: Christopher Griffin christophergriffin05@gmail.com 🤖 Tested and written up with Claude Code |
|
Gentle ping: this fixes the long-standing @robimarko, @Ansuel, could one of you take a look and merge it if it looks good? A follow-up for small ring sizes (#11) is stacked on this one, so it will need a rebase after this merges. Thanks! |
logd keeps the messages in a ring buffer. When a message does not fit before the end of the buffer, log_add() writes an end marker (an entry of size 0) where the current round ends and starts over at the beginning, and "oldest" then points into the entries of the previous round. As the new round grows, log_add() pushes "oldest" forward through those entries, but nothing stops it at the end marker: the entry after a zero-sized one is its header's own end, which is stale data of an earlier round. From then on log_list() starts from garbage, so "logread" gets entries with random sizes and timestamps, and logd can read past the end of the buffer. musl's asctime_r() aborts on such a timestamp, which shows up as traps: logread[...] general protection fault ... in libc.so Messages of varying length make it likely, e.g. "logger" with 1 to 100 characters in a loop next to "logread -l 1" in a loop (FS#4118). Stop at the end marker: when "oldest" reaches it, the oldest entry is the first one of the new round, at the start of the buffer. Also start over at the beginning when the header of the next entry does not fit before the end of the buffer, not only when its first byte does not: an entry that starts in the last bytes of the buffer has its header cut, and log_list() reads past the end of the buffer there. Assisted-by: Claude Code (Claude Opus 5.5) Signed-off-by: Pierre-Yves Le Borgne <pylaterreur@gmail.com> Link: openwrt#9 Signed-off-by: Robert Marko <robimarko@gmail.com>
cddf6c5 to
46f702b
Compare
|
Thanks! Rebased on top of master and merged! |
thanks! #11 rebased on origin now :) |
This fixes the crashes of FS#4118 (openwrt/openwrt#9100), "logread crashes sometimes", open since 2021.
logd keeps its messages in a ring buffer. When a message doesn't fit before the end of the buffer,
log_add()writes an end marker (an entry of size 0) and starts over at the beginning, andoldestthen points into the previous round. As the new round grows,log_add()pushesoldestforward through the previous round's entries. Nothing stops it at the end marker, though:log_next()of a zero-sized entry is the byte right after its header, i.e. stale data of an earlier round. From then onlog_list()starts from garbage.logreadthen gets entries with random sizes and timestamps, and logd can read past the end of its buffer. On musl,asctime_r()aborts on such a timestamp, which shows up astraps: logread[...] general protection fault ... in libc.so. glibc prints a garbage date instead of aborting, but logd still reads past the end of its buffer.The fix stops at the end marker: when
oldestreaches it, the oldest entry is the first one of the new round, at the start of the buffer. It also wraps when the header of the next entry would not fit before the end. Before, it wrapped only when the entry's first byte would not fit, solog_list()could read a cut header past the end of the buffer.Tested:
log_next(), the ring part oflog_add()andlog_list()into a test that adds 100,000 messages of 2 to 1024 bytes into 16, 64 and 128 KiB buffers (8 seeds each), under ASan. After each add, it walks the entries for-l 2,-l 20and a full read, and checks each entry's size, id order and timestamp.log_list().loggerwith 100 down to 1 characters in a loop, next tologread -l 2in loops (plus full reads andlogread -f). With master, logd hit a heap-buffer-overflow read (strleninlog_fill_msg()<-read_log()) within 90 s on a 16 KiB ring. With this commit, 90 s ran without any ASan report or a bad date (13,800logread -l 2runs, 13,400 messages).logread -l 2in two loops, full and-ereads, shortlogread -fruns and aloggerflood gave 3logreadtraps. With this commit, a 30-minute run (26,583logread -l 2) gave none.🤖 Generated with Claude Code
https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN