Skip to content

logd: don't hand out stale entries after the ring wraps - #9

Merged
openwrt-bot merged 1 commit into
openwrt:masterfrom
pylaterreur:logd-ring-walk
Nov 15, 2026
Merged

openwrt-bot merged 1 commit into
openwrt:masterfrom
pylaterreur:logd-ring-walk

Conversation

@pylaterreur

@pylaterreur pylaterreur commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

This fixes the crashes of FS#4118 (openwrt/openwrt#9100), "logread crashes sometimes", open since 2021.

logd keeps its messages in a ring buffer. When a message doesn't fit before the end of the buffer, log_add() writes an end marker (an entry of size 0) and starts over at the beginning, and oldest then points into the previous round. As the new round grows, log_add() pushes oldest forward through the previous round's entries. Nothing stops it at the end marker, though: log_next() of a zero-sized entry is the byte right after its header, i.e. stale data of an earlier round. From then on log_list() starts from garbage.

logread then gets entries with random sizes and timestamps, and logd can read past the end of its buffer. On musl, asctime_r() aborts on such a timestamp, which shows up as traps: logread[...] general protection fault ... in libc.so. glibc prints a garbage date instead of aborting, but logd still reads past the end of its buffer.

The fix stops at the end marker: when oldest reaches it, the oldest entry is the first one of the new round, at the start of the buffer. It also wraps when the header of the next entry would not fit before the end. Before, it wrapped only when the entry's first byte would not fit, so log_list() could read a cut header past the end of the buffer.

Tested:

  • Simulation. I copied log_next(), the ring part of log_add() and log_list() into a test that adds 100,000 messages of 2 to 1024 bytes into 16, 64 and 128 KiB buffers (8 seeds each), under ASan. After each add, it walks the entries for -l 2, -l 20 and a full read, and checks each entry's size, id order and timestamp.
    • Before: a bad entry within the first ~2,000 to 5,000 adds of every seed.
    • With only the end-marker change: a heap-buffer-overflow read in log_list().
    • With both changes: no bad walk in the 24 runs.
  • Host, ASan. ubusd, logd and logread from libubox e7608b6, ubus 24864e7 and this tree, built with ASan, ran bittorf's reproducer from FS#4118: logger with 100 down to 1 characters in a loop, next to logread -l 2 in loops (plus full reads and logread -f). With master, logd hit a heap-buffer-overflow read (strlen in log_fill_msg() <- read_log()) within 90 s on a 16 KiB ring. With this commit, 90 s ran without any ASan report or a bad date (13,800 logread -l 2 runs, 13,400 messages).
  • OpenWrt 328acd4e, x86_64, QEMU (OpenMPTCProuter build, ubox 6f78fa4). 20 minutes of logread -l 2 in two loops, full and -e reads, short logread -f runs and a logger flood gave 3 logread traps. With this commit, a 30-minute run (26,583 logread -l 2) gave none.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN

WiFiGUY512 added a commit to WiFiGUY512/openwrt-arista that referenced this pull request Nov 6, 2026
…(B162)

openwrt/ubox#9 as a ubox package patch (r2): fixes the occasional logread
SIGSEGV (FS#4118 / openwrt#9100) seen in the C-360s' logread pipelines.
W-318 reproducer: stock 6 crashes in 150 s, patched 0 in 300 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WiFiGUY512

Copy link
Copy Markdown

Tested on real hardware: Arista W-318 (Qualcomm IPQ6018, aarch64, musl), OpenWrt main with ubox 6f78fa4, 128 KiB ring (log_size 128).

FS#4118 reproducer as in your description: logger with 100 down to 1 characters in a loop, next to two logread -l 2 loops and a full logread loop.

  • Stock logd: 6 logread SIGSEGVs in 150 s. Each core shows musl's asctime_r() hitting a_crash() on a garbage time ("Mon Jan 15 19:33:40 34385", "Fri Apr 22 05:47:59 81499"), from ctime() in log_notify() (logread.c:169) <- logread_fd_data_cb().
  • logd with this PR: 0 crashes in 300 s (79,598 logread -l 2 runs, 4,772 full reads, 51,900 messages).

We hit it in the field first, on IPQ8074 APs: twice in nine days, in logread | grep ... pipelines that run every few minutes.

Tested-by: Christopher Griffin christophergriffin05@gmail.com

🤖 Tested and written up with Claude Code

1 similar comment
@WiFiGUY512

Copy link
Copy Markdown

Tested on real hardware: Arista W-318 (Qualcomm IPQ6018, aarch64, musl), OpenWrt main with ubox 6f78fa4, 128 KiB ring (log_size 128).

FS#4118 reproducer as in your description: logger with 100 down to 1 characters in a loop, next to two logread -l 2 loops and a full logread loop.

  • Stock logd: 6 logread SIGSEGVs in 150 s. Each core shows musl's asctime_r() hitting a_crash() on a garbage time ("Mon Jan 15 19:33:40 34385", "Fri Apr 22 05:47:59 81499"), from ctime() in log_notify() (logread.c:169) <- logread_fd_data_cb().
  • logd with this PR: 0 crashes in 300 s (79,598 logread -l 2 runs, 4,772 full reads, 51,900 messages).

We hit it in the field first, on IPQ8074 APs: twice in nine days, in logread | grep ... pipelines that run every few minutes.

Tested-by: Christopher Griffin christophergriffin05@gmail.com

🤖 Tested and written up with Claude Code

@pylaterreur

Copy link
Copy Markdown
Contributor Author

Gentle ping: this fixes the long-standing logread crashes of FS#4118 (openwrt/openwrt#9100) and has been open since 2026-10-07. It was also tested on real hardware by another user (comment above: 6 logread SIGSEGVs in 150 s on stock logd, none in 300 s with this change).

@robimarko, @Ansuel, could one of you take a look and merge it if it looks good? A follow-up for small ring sizes (#11) is stacked on this one, so it will need a rebase after this merges. Thanks!

logd keeps the messages in a ring buffer. When a message does not fit
before the end of the buffer, log_add() writes an end marker (an entry
of size 0) where the current round ends and starts over at the
beginning, and "oldest" then points into the entries of the previous
round. As the new round grows, log_add() pushes "oldest" forward
through those entries, but nothing stops it at the end marker: the
entry after a zero-sized one is its header's own end, which is stale
data of an earlier round. From then on log_list() starts from garbage,
so "logread" gets entries with random sizes and timestamps, and logd
can read past the end of the buffer. musl's asctime_r() aborts on such
a timestamp, which shows up as

  traps: logread[...] general protection fault ... in libc.so

Messages of varying length make it likely, e.g. "logger" with 1 to 100
characters in a loop next to "logread -l 1" in a loop (FS#4118).

Stop at the end marker: when "oldest" reaches it, the oldest entry is
the first one of the new round, at the start of the buffer.

Also start over at the beginning when the header of the next entry does
not fit before the end of the buffer, not only when its first byte does
not: an entry that starts in the last bytes of the buffer has its header
cut, and log_list() reads past the end of the buffer there.

Assisted-by: Claude Code (Claude Opus 5.5)
Signed-off-by: Pierre-Yves Le Borgne <pylaterreur@gmail.com>
Link: openwrt#9
Signed-off-by: Robert Marko <robimarko@gmail.com>
@openwrt-bot
openwrt-bot merged commit 46f702b into openwrt:master Nov 15, 2026
@robimarko

Copy link
Copy Markdown

Thanks! Rebased on top of master and merged!

@pylaterreur

Copy link
Copy Markdown
Contributor Author

Thanks! Rebased on top of master and merged!

thanks! #11 rebased on origin now :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants