Skip to content

benchmark: give the database tests their own category and CPU split - #1715

Open
MDA2AV wants to merge 2 commits into
mainfrom
database-category
Open

MDA2AV wants to merge 2 commits into
mainfrom
database-category

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Follows the discussion in #1482.

What changes

  • Own category. async-db and fortunes move out of HTTP/1.1 into a new Database family (db), at the same level as H/2 and H/3. It gets its own composite, nav entry, detailed views, badges, pre-rendered pages and row in the PR composite table.

  • Only async-db is scored for now, on this board and nowhere else. The driver dominating its number rules it out of a ranking of HTTP paths, and is what this board ranks. fortunes stays reference-only, as a faded column, because 63 of the 101 entries running async-db have no Fortunes and scoring it would cap all of them at half the board.

  • Its own CPU split. They still run over HTTP/1.1 with gcannon, but the box is split three ways, with every core paired with its SMT sibling (N and N+64):

    Role Logical CPUs Cpuset Before
    Server 32 0-15,64-79 0-31,64-95
    Load generator 32, with 32 threads 16-31,80-95 32-63,96-127, 64 threads
    Postgres 64 32-63,96-127 unpinned, on top of the other two
  • Postgres only stays pinned for these tests. It's moved with docker update, and the gateway profiles release it, so they run as before.

  • Docs move to test-profiles/database/. The old URLs aren't redirected, the same as earlier docs moves.

The cpusets are overridable (DB_SERVER_CPUS, DB_LOADGEN_CPUS, DB_POSTGRES_CPUS, DB_THREADS). On a box too small for them, the generator and Postgres stay where they are.

How I read the request

"32 CPU" means 32 logical CPUs, so 16 cores plus siblings. 32 cores each would take the whole box and leave nothing for Postgres.

Impact

  • HTTP/1.1 rankings don't change. Recomputing all four leagues with both tests moved out changed 0 ranks and 0 scores, because they were reference-only there.
  • The new board starts from numbers measured on the old split, until the re-run below. On those numbers, flagship and emerging are led by dusty-tuned (1000), cpeak (980) and genhttp-ioxide (910).
  • Medals: async-db was reference-only for frameworks, so it awarded them none. It does now, so medal counts near the top go up.

Before merging

  • Try the split on the bench host. A /benchmark comment runs this PR's scripts, and results compare against the published numbers, so expect them to move: the server has half the CPUs, and Postgres has its own.
    /benchmark-multiple -f aspnet-minimal,genhttp-ioxide,actix,helidon-production,fiber -t async-db,fortunes
    
  • Plan the re-run after merge, so the board isn't mixing the old and new splits: /benchmark-test -t async-db and /benchmark-test -t fortunes.
  • Whichever of this and benchmark: warm up on every profile, 2x15s on the main throughput ones #1714 merges second has small conflicts to resolve, in scripts/lib/profiles.sh and test-profiles/_index.md.

Verified

On the dev box, which has 32 logical CPUs, so the split was overridden to server 0-7, Postgres 8-15 and generator 16-31 with 16 threads:

Check Result
aspnet-minimal async-db,fortunes through benchmark.sh --save server 0-7 and Postgres 8-15 from docker inspect, 16 threads in the result rows
The running generator on async-db affinity 16-31, -t 16
postgres_pin pin and release Postgres PID 1's Cpus_allowed_list goes 8-15, then 0-31
Default split on this small box warns and falls back, nothing pinned
profiles.sh sourced alone under set -u, the way validate.sh does works
gen_leaderboard_data.py and check_badge_parity.js badge parity ok, 1611 ranks match, including the new family
Board and docs in headless Chrome Database board, its detailed views and docs render, with Fortunes faded, and the HTTP/1.1 board no longer shows the two columns

Not run: the real split on a 128-CPU box, and a gateway profile straight after a database one. Only postgres_pin's release was tested on its own.

🤖 Generated with Claude Code

async-db and fortunes move out of HTTP/1.1 into a Database family at the same
level as H/2 and H/3, with their own composite, scored there and nowhere else. The
driver dominating these numbers rules them out of a ranking of HTTP paths, and is
what this board ranks.

They still run over HTTP/1.1 with gcannon, but split the box three ways because
Postgres does real work on them: 32 logical CPUs for the server (0-15,64-79), 32
for the generator with 32 threads (16-31,80-95) and the rest for Postgres
(32-63,96-127), each as cores plus their SMT siblings. Postgres used to float over
the server's and the generator's CPUs. The gateway profiles release it again, so
they run as before.

The docs move to test-profiles/database/.
Fortunes goes back to reference-only, shown as a faded column. 63 of the 101
entries running async-db have no Fortunes, so scoring it would cap all of them at
half the board.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant