Skip to content

fix: create the HzMemoryCache ActivitySource once instead of per call - #28

Merged
johan-ketels merged 3 commits into
mainfrom
fix-activitysource-per-call-allocation
Sep 7, 2026
Merged

johan-ketels merged 3 commits into
mainfrom
fix-activitysource-per-call-allocation

Conversation

@johan-ketels

Copy link
Copy Markdown
Contributor

Problem

HzActivities.Source is an expression-bodied property:

public static ActivitySource? Source => new(HzCacheActivitySourceName);

Every access constructs a new ActivitySource. It is accessed from 27 call sites (HzMemoryCache, HzCacheMemoryLocker, RedisBackedHzCache), and the cleanup timer alone (cleanupJobInterval default 1000 ms) accesses it twice per tick via ProcessExpiredEviction and EvictExpired, so a process creates at least 2 new sources per second even when the cache is idle.

Every ActivitySource constructor registers the instance in the runtime's static s_activeSources list and only Dispose() removes it, which never happens here. Since .NET 9 that list is a copy-on-write array: each registration allocates an array of n+1 references and copies the previous n. So the cost of each creation grows with process uptime, the arrays land on the LOH, and gen2 collections follow.

Observed effect in admin-backend (Admin.API, .NET 10)

RC pod at idle (about one request per hour, zero SQL operations):

Metric Pod start 2.5 days later
Allocation rate 0.3 MB/s 7.1 MB/s
Gen2 collections 16 / h 302 / h
Gen0 collections 31 / h 31 / h
Gen2 heap 98 MiB 153 MiB
CPU 0.029 cores 0.050 cores

At 2 creations/s the copy-on-write model predicts 6.9 MB/s after 2.5 days and about 55 MiB of retained sources; observed 7.1 MB/s and 55 MiB. Busier environments (stage, prod) climb faster because request-path cache operations add more creations. Memory and CPU grow until the next deploy resets the process.

Fix

A single static instance, which is the documented usage pattern for ActivitySource:

public static readonly ActivitySource Source = new(HzCacheActivitySourceName);

No call-site changes needed. The type is no longer nullable, which only removes a nullable warning at the call sites.

Verification

  • dotnet build hzcache.sln -c Release: 0 errors
  • dotnet test --filter "TestCategory!=Integration" (same filter as CI): 36 passed, 0 failed

Follow-up

  • Publish as release/0.0.19 and bump RedisBackedHzCache in commerce-admin (src/Infrastructure, currently 0.0.18) and Storm.Admin (0.0.16).
  • Norce.Data.Sharding.Common.Instrumentation.Source in data-sharding has the same pattern at a much lower call rate; same one-line fix.

🤖 Generated with Claude Code

johan-ketels and others added 2 commits September 7, 2026 09:33
HzActivities.Source was an expression-bodied property, so every cache
operation and every cleanup-timer tick constructed a new ActivitySource.
Each instance registers itself in the runtime's global source list and is
never disposed, so the list grows for the lifetime of the process.

Since .NET 9 that list is a copy-on-write array: each registration
allocates a new array and copies all previous entries. In admin-backend
this showed up as allocation rate and gen2 GC rate climbing linearly with
uptime while idle (0.3 -> 7 MB/s over 2.5 days at ~2 creations/s from the
1 s cleanup timer alone), with memory and CPU following until the next
deploy.

A single static instance is the documented usage pattern for
ActivitySource and removes the growth entirely.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Asserts that HzActivities.Source is a single instance and that a burst of
cache operations plus cleanup-timer ticks registers no new ActivitySource,
observed through an ActivityListener's ShouldListenTo callback. Both tests
fail against the previous expression-bodied property.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@johan-ketels
johan-ketels marked this pull request as ready for review September 7, 2026 08:51
@johan-ketels
johan-ketels requested a review from a team as a code owner September 7, 2026 08:51
Copilot AI lite review requested due to automatic review settings September 7, 2026 08:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It introduces a binary-breaking public API change (Source property → field) and the new test has a cross-thread data race on the armed flag that can yield false negatives.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR addresses a runtime/performance issue where HzActivities.Source was constructing a new ActivitySource on every access, causing steadily increasing allocation and CPU cost over process uptime due to repeated global registration.

Changes:

  • Replace per-access ActivitySource construction with a single shared instance in HzActivities.
  • Add unit tests to assert HzActivities.Source is stable and that cache operations do not create additional ActivitySource instances.
File summaries
File Description
UnitTests/DiagnosticsTests.cs Adds regression tests to detect per-call ActivitySource creation.
HzMemoryCache/Diagnostics/HzActivities.cs Switches HzActivities.Source to a single shared instance to avoid repeated registrations/allocations.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread HzMemoryCache/Diagnostics/HzActivities.cs
Comment thread UnitTests/DiagnosticsTests.cs Outdated
Expose the single instance through a property backed by a private static
field, so consumers compiled against get_Source() keep working.

The test now captures the expected instance before registering the
listener and counts any other source with the same name, which removes the
cross-thread flag and the dependence on which test touches Source first.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 7, 2026 09:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is small, targeted, and includes focused unit tests that directly prevent regression of the underlying allocation issue.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@johan-ketels
johan-ketels merged commit 5e6ec09 into main Sep 7, 2026
2 checks passed
@johan-ketels
johan-ketels deleted the fix-activitysource-per-call-allocation branch September 7, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants