Skip to content

benchmark: add an Interaction category with a Rendezvous test - #1717

Open
MDA2AV wants to merge 1 commit into
mainfrom
rendezvous
Open

MDA2AV wants to merge 1 commit into
mainfrom
rendezvous

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Refs #1686.

The test

GET /rendezvous/{key}?v={value}

The first request for a key waits. The second completes both, and each is answered with the other's value. So no response can be produced from one request alone.

Every other profile answers a request from that request alone, which is exactly what one shared-nothing event loop per core is built for. Here the two sides usually sit on different workers, so something has to cross between them on every exchange: a queue, a wakeup, a lock, a copy or a message to another process. That is the cost the issue says the board never shows.

Why this shape, given the bar set in the issue, that it must run on every technology and compare apples to apples:

  • Any entry with async handlers can do it. It's plain HTTP/1.1, so it isn't limited to WebSocket entries, and most handlers are 20-40 lines.
  • There's nothing to compare unfairly. No database, driver, ORM, cache or template, which is what made CRUD hard to balance. Only the runtime's own coordination primitives.
  • No new load generator. It's wrk with a Lua script.
  • A server can't arrange its own pairs. Every two consecutive requests on a generator thread share a key, on whichever connections happen to be free, so the server only learns who is paired once both have sent.

What's in this PR

  • A new Interaction category (ix), at the same level as H/2 and H/3, with its own board, badges and docs. Rendezvous is scored there and counts toward no other composite.

  • The rendezvous profile, 4,096 connections, 3×5s, with requests/rendezvous.lua.

  • scripts/validate-rendezvous.py, which is the whole anti-cheat, because wrk doesn't read the bodies.

  • Six entries, one for each way of running a server:

    Entry Model How it waits
    actix one worker per core DashMap and a tokio oneshot
    vertx one event loop per core ConcurrentHashMap, answered back on the waiter's event loop
    aspnet-minimal thread pool ConcurrentDictionary and a TaskCompletionSource
    chi goroutines sync.Map and a channel
    helidon-production virtual threads ConcurrentHashMap and a blocking future
    node one process per core pairing in the primary, every request crossing IPC

First numbers

Local only, so directional. It's a 14900K with the server on 16 P-core threads and wrk on the 16 E-cores, best of 3×5s. The plain GET uses the same generator, connections and pinning, so the difference is the crossing and not the tool.

Entry Plain GET req/s Rendezvous req/s Kept CPU per request
actix 2,232,777 2,134,348 96% 6.0us → 7.4us (+24%)
aspnet-minimal 1,344,679 1,237,843 92% 11.4us → 12.3us (+9%)
helidon-production 1,818,364 1,558,432 86% 8.7us → 10.2us (+17%)
vertx 2,148,249 1,817,198 85% 6.9us → 8.7us (+27%)
chi 1,096,390 913,663 83% 14.2us → 17.1us (+20%)
node 1,459,005 458,896 31% 10.9us → 18.9us (+73%)
  • One process per core pays by far the most. node keeps 31%, because every request goes through the primary over IPC.
  • Threads pay much less than the issue expects: 4-17% of throughput and 9-27% more CPU per request. The CPU column is the fairer one, since actix wasn't saturating the server on its plain GET (about 13.4 of 16 cores), so its 96% flatters it.
  • The issue's claim is neither confirmed nor refuted yet. actix, the closest thing here to the shared-nothing designs it describes, kept more than chi. But no thin C, C++ or Zig engine implements this yet, and that's the case the issue is about. 16 CPUs may also understate it, because a shared map contends harder on the bench host's 64.

Verified

  • All six entries pass validate.sh, including the 8 new checks.

  • The validator fails servers that break the contract. I ran it against throwaway servers:

    Server Result
    Correct 8 passed
    Echoes its own value 6 failed
    Answers without waiting 6 failed
    Pairs any two requests, ignoring the key 2 failed
    Never frees a key 1 failed
    A map per worker process 4 failed
  • wrk reported no socket errors on any rendezvous run. A pair stuck without a partner would show up as timeouts.

  • benchmark.sh --save works end to end on actix and node. The site builds with and without results, badge parity passes, and the board and docs render in headless Chrome.

Not run: the bench host.

Before merging

🤖 Generated with Claude Code

GET /rendezvous/{key}?v={value}: the first request for a key waits, the second
completes both, and each is answered with the other's value. Every other profile
answers a request from that request alone, which is the case one shared-nothing
event loop per core is built for. Here the two sides usually sit on different
workers, so something has to cross between them on every exchange.

It runs over plain HTTP with wrk and a Lua script that pairs requests as they go
out, so a server can't arrange for a pair to land on one worker, and any entry with
async handlers can subscribe. It's scored on its own board and counts toward no
other composite.

chi, actix, aspnet-minimal, vertx, helidon-production and node implement it, one
each of goroutines, one worker per core, a thread pool, one event loop per core,
virtual threads and one process per core.

Refs #1686
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

👋 Heads up! This PR modifies the following frameworks:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant