Skip to content

Test the workerd collector's memory, queue and retention bounds - #6

Merged
RhysSullivan merged 2 commits into
mainfrom
test/bounded-collector-memory
Sep 28, 2026
Merged

RhysSullivan merged 2 commits into
mainfrom
test/bounded-collector-memory

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Sep 27, 2026 •

Copy link
Copy Markdown

Runs the built worker as a real workerd process with its inspector open.

  • Sustained ingest (36,000 spans at up to 400/s over 90 s): buffers the collector's isolate holds stay within maxPending × maxBytes + 8 MiB. The V8 heap, sampled without forcing a collection, stays at or below 64 MiB. Live heap after a collection stays under 32 MiB. After the retention pass the store holds at most the configured spans and bytes.
  • With only the size bound in reach and no retention pass during the load, the database file never exceeds the bound by more than one export, and the stored bytes end within it.
  • With the queue full of exports whose bodies are still arriving, the next export gets 429 and an oversized one 413; both are counted in GET /api/ingest, and the queue frees once the stalled exports finish.
  • Malformed exports get 400 and are counted as invalid. The counts are the same after the collector restarts on the same store.

Results of this test file on each collector revision:

revision heap / buffers file size bound queue loss counts
edca7f8 (base, #2) fail: 294 MiB buffers vs 12 fail: 34 MB stored vs 4 MiB fail: 200, nothing refused fail: 500 for malformed
2f5bd17 (router only) fail: heap 92.8 MiB vs 64 fail fail (new counters) fail
89fbfd3 (direct ingest) pass fail: 34 MB stored vs 4 MiB fail (new counters) fail
c9422c3 (#5 head) pass pass pass pass

bun run test on #5's head with these tests: 217 pass, 0 fail.

@RhysSullivan
RhysSullivan force-pushed the test/bounded-collector-memory branch from e809bb8 to 25cabff Compare September 27, 2026 18:41
@RhysSullivan
RhysSullivan changed the base branch from fix/bounded-collector-memory to main September 28, 2026 13:13
@RhysSullivan
RhysSullivan marked this pull request as ready for review September 28, 2026 13:13
@RhysSullivan
RhysSullivan merged commit 9eeda59 into main Sep 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant