Skip to content

Write initialized buffers back so read-only tests stay read-only - #62

Open
superworm wants to merge 1 commit into
google:masterfrom
superworm:clean-readonly-arenas
Open

superworm wants to merge 1 commit into
google:masterfrom
superworm:clean-readonly-arenas

Conversation

@superworm

Copy link
Copy Markdown
Contributor

multiload and multichase initialize their buffers with stores (memset of the load buffer, writing the chase pointers), so a read-only test starts with every line dirty. On system level caches where a read hit on a dirty line returns the line dirty and drops the cached copy (Neoverse N4 "Cedar" gMesh SLC, for example), every pass of stream-sum or a pointer chase that fits in the SLC then writes each line back: one SLC re-allocation per hit, and write bandwidth equal to read bandwidth on a read-only test. The measured SLC hit rate drops to 50%.

The -F flush does not prevent this: it reads a buffer that was itself filled with memset, only chase threads run it, and a 64 MiB flush is smaller than the last level cache of current server parts.

Add clean_invalidate_range() to cpu_util.h. On aarch64 it issues DC CIVAC over the range (line size from CTR_EL0) followed by DSB ISH. DC CIVAC is ARMv8.0 and usable from user space on Linux. On other architectures it is a no-op and HAVE_CLEAN_INVALIDATE_RANGE is 0, so behavior there is unchanged except as noted below.

  • multiload: clean the load buffer after memset.
  • multiload and multichase chase setup: on aarch64, a non-zero -F now writes the chase back with clean_invalidate_range() instead of reading the flush buffer, and the flush buffer is not allocated.
  • multiload: do not allocate the flush buffer in bandwidth-only runs on any architecture. Only chase threads read it, so it only added 64 MiB of dirty lines.

Neoverse N4, one thread on cpu 8 (two for chaseload), 20 MiB buffer, chases with -L -H -T 2m, SLC PMU over 3 s of steady state, before -> after:

test slc_exec_hit slc_exec_alloc_req latency
multiload stream-sum 1353M -> 2697M 1345M -> 1M
multiload chase 110M -> 215M 106M -> 1M 31.96 ns both
multiload chaseload 1683M -> 2920M 1236M -> 4M
multichase chase 111M -> 217M 104M -> 1M 31.96 ns both

An idle machine shows about 0.5M allocations over the same window. Hit counts double because once lines are clean the SLC keeps its copy and the core's clean eviction of the line also counts as a hit. Chase latency is unchanged.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a clean_invalidate_range function on aarch64 architectures to write back and invalidate a memory range to the point of coherency using the DC CIVAC instruction. This is integrated into multichase and multiload to ensure that benchmarks start with clean, uncached buffers rather than dirty cache lines, avoiding unnecessary write traffic. The review feedback correctly identifies a bug in clean_invalidate_range where a length of zero with an unaligned buffer pointer can cause the loop to execute once, potentially leading to unexpected cache invalidation or segmentation faults. A guard check to return early when len == 0 is recommended.

Comment thread cpu_util.h
multiload and multichase initialize their buffers with stores (memset of
the load buffer, writing the chase pointers), so a read-only test starts
with every line dirty. On system level caches where a read hit on a
dirty line returns the line dirty and drops the cached copy (Neoverse N4
"Cedar" gMesh SLC, for example), every pass of stream-sum or a pointer
chase that fits in the SLC then writes each line back: one SLC
re-allocation per hit, and write bandwidth equal to read bandwidth on a
read-only test. The measured SLC hit rate drops to 50%.

The -F flush does not prevent this: it reads a buffer that was itself
filled with memset, only chase threads run it, and a 64 MiB flush is
smaller than the last level cache of current server parts.

Add clean_invalidate_range() to cpu_util.h. On aarch64 it issues DC CIVAC
over the range (line size from CTR_EL0) followed by DSB ISH. DC CIVAC is
ARMv8.0 and usable from user space on Linux. On other architectures it is
a no-op and HAVE_CLEAN_INVALIDATE_RANGE is 0, so behavior there is
unchanged except as noted below.

 - multiload: clean the load buffer after memset.
 - multiload and multichase chase setup: on aarch64, a non-zero -F now
   writes the chase back with clean_invalidate_range() instead of reading
   the flush buffer, and the flush buffer is not allocated.
 - multiload: do not allocate the flush buffer in bandwidth-only runs on
   any architecture. Only chase threads read it, so it only added 64 MiB
   of dirty lines.

Neoverse N4, one thread on cpu 8 (two for chaseload), 20 MiB buffer,
chases with -L -H -T 2m, SLC PMU over 3 s of steady state, before ->
after:

  test                 slc_exec_hit     slc_exec_alloc_req   latency
  multiload stream-sum 1353M -> 2697M   1345M -> 1M
  multiload chase       110M ->  215M    106M -> 1M          31.96 ns both
  multiload chaseload  1683M -> 2920M   1236M -> 4M
  multichase chase      111M ->  217M    104M -> 1M          31.96 ns both

An idle machine shows about 0.5M allocations over the same window. Hit
counts double because once lines are clean the SLC keeps its copy and the
core's clean eviction of the line also counts as a hit. Chase latency is
unchanged.
@superworm
superworm force-pushed the clean-readonly-arenas branch from 59d56d2 to d8202a0 Compare October 16, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant