From 9ce0b64238c2002bf43adb7039da8077fc6eaed0 Mon Sep 17 00:00:00 2001 From: MDA2AV Date: Thu, 5 Nov 2026 23:14:30 +0000 Subject: [PATCH 1/2] benchmark: give the database tests their own category and CPU split async-db and fortunes move out of HTTP/1.1 into a Database family at the same level as H/2 and H/3, with their own composite, scored there and nowhere else. The driver dominating these numbers rules them out of a ranking of HTTP paths, and is what this board ranks. They still run over HTTP/1.1 with gcannon, but split the box three ways because Postgres does real work on them: 32 logical CPUs for the server (0-15,64-79), 32 for the generator with 32 threads (16-31,80-95) and the rest for Postgres (32-63,96-127), each as cores plus their SMT siblings. Postgres used to float over the server's and the generator's CPUs. The gateway profiles release it again, so they run as before. The docs move to test-profiles/database/. --- README.md | 6 ++-- scripts/benchmark.sh | 15 +++++++++ scripts/check_badge_parity.js | 2 +- scripts/gen_leaderboard_data.py | 33 ++++++++++--------- scripts/lib/postgres.sh | 8 +++++ scripts/lib/profiles.sh | 24 ++++++++++++-- scripts/validate.sh | 4 +-- site/content/docs/add-framework/badges.md | 3 +- .../docs/running-locally/configuration.md | 4 +++ .../docs/running-locally/scripts/benchmark.md | 6 +++- site/content/docs/scoring/achievements.md | 2 +- site/content/docs/scoring/composite-score.md | 15 ++++++--- site/content/docs/test-profiles/_index.md | 3 +- .../docs/test-profiles/database/_index.md | 23 +++++++++++++ .../{h1 => database}/async-database/_index.md | 0 .../async-database/implementation.md | 10 +++--- .../async-database/validation.md | 0 .../{h1 => database}/fortunes/_index.md | 2 +- .../fortunes/implementation.md | 4 ++- .../{h1 => database}/fortunes/validation.md | 0 .../gateway/gateway-h2/implementation.md | 2 +- .../gateway/gateway-h2/validation.md | 4 +-- site/content/docs/test-profiles/h1/_index.md | 4 +-- site/leaderboard/index.html | 10 +++--- 24 files changed, 134 insertions(+), 50 deletions(-) create mode 100644 site/content/docs/test-profiles/database/_index.md rename site/content/docs/test-profiles/{h1 => database}/async-database/_index.md (100%) rename site/content/docs/test-profiles/{h1 => database}/async-database/implementation.md (93%) rename site/content/docs/test-profiles/{h1 => database}/async-database/validation.md (100%) rename site/content/docs/test-profiles/{h1 => database}/fortunes/_index.md (88%) rename site/content/docs/test-profiles/{h1 => database}/fortunes/implementation.md (95%) rename site/content/docs/test-profiles/{h1 => database}/fortunes/validation.md (100%) diff --git a/README.md b/README.md index e05a8218cf..c2bd1e229e 100644 --- a/README.md +++ b/README.md @@ -47,8 +47,7 @@ The reply states which baseline it used, and profiles the other framework does n | Concurrency | `async` | `GET /delay/{ms}` with the wait named in the route, 32K held connections. Isolates what a framework does while a request is pending, with no database or I/O in the way. Scored for frameworks and engines, not for infrastructure, since the spec settled at a flat 10 ms over 32K connections | | Efficiency | `latency-1m`, `latency-10k`, `latency-500k-8cpu` * | One million req/s offered at a fixed rate with zrk; the metric is CPU spent per request, read exactly from the container's cgroup rather than sampled. Everyone who finishes serves the same million, so what separates them is the bill. Scored on its own 0-100 formula rather than on rps, since the rate is pinned. `latency-10k` is the identical setup - same cores, connections and endpoint - at a near-idle 10K req/s, so what it reports is the standing cost of being a running server rather than of serving load, and it is scored the same way. Both score CPU at 0.50, p99 at 0.25 and mean latency at 0.25, the mean being the queueing signal: ten times the best mean is worth nothing. `latency-500k-8cpu` pins 500K req/s with the server on four cores plus their SMT siblings, so the load sits near saturation and the queue a server builds shows in its mean; reference-only while the first entries are measured | | Workload | `json-comp`, `json-tls`, `8gbit`, `static-tls` | JSON serialization, gzip/brotli compression, HTTP/1.1 over TLS, a 10 KB body echoed over TLS at a pinned 50K req/s so ingest and egress load together - scored on the same 0-100 formula as the latency profiles, since its rate is pinned too, 20-file static asset serving over TLS | -| Database | `async-db` * | Async Postgres sequential scan. Reference-only: the database and its driver dominate it far more than the framework does, so it is measured and shown but no longer decides the ranking | -| Templates | `fortunes` * | DB query + HTML template render (TechEmpower-style Fortunes). Reference-only: measures template-engine throughput, and is not part of the composite score | +| Database | `async-db`, `fortunes` | Postgres-backed HTTP/1.1, on their own board: an async range query, and a DB query + HTML template render (TechEmpower-style Fortunes). The server gets 32 logical CPUs, the load generator 32 and Postgres the rest | | H/2 | `baseline-h2`, `static-h2`, `baseline-h2c`, `json-h2c` | Baseline + static over TLS with h2 stream multiplexing; baseline + JSON over cleartext h2 (prior-knowledge, port 8082) | | H/3 | `baseline-h3`, `static-h3` | Baseline and static over QUIC with TLS 1.3 | | gRPC | `unary-grpc`, `unary-grpc-tls` | Unary gRPC over plaintext HTTP/2 and TLS | @@ -99,12 +98,13 @@ Swap `actix` for your entry and `h1` for the family you want: | Family | `` | Composite it reports | |---|---|---| -| HTTP/1.1 | `h1` | Connection, Workload, Database and Multi-endpoint profiles | +| HTTP/1.1 | `h1` | Connection, Workload and Multi-endpoint profiles | | HTTP/2 | `h2` | the HTTP/2 profiles | | HTTP/3 | `h3` | the HTTP/3 profiles | | Gateway | `gw` | reverse-proxy and production-stack profiles | | gRPC | `grpc` | unary and streaming, plaintext and TLS | | WebSocket | `ws` | the echo profiles | +| Database | `db` | the Postgres-backed profiles | `` is your entry's name on the board, lowercased, with anything outside `A-Z a-z 0-9 . _ -` turned into `-`, the same name as your file in diff --git a/scripts/benchmark.sh b/scripts/benchmark.sh index 642846464b..b3e5872089 100755 --- a/scripts/benchmark.sh +++ b/scripts/benchmark.sh @@ -227,6 +227,10 @@ run_one() { banner "$FRAMEWORK / $profile / ${CONNS}c (tool=$tool)" + # Local so the adapters and save_result() see the database split below, and + # the next profile does not. + local GCANNON_CPUS="$GCANNON_CPUS" THREADS="$THREADS" + # Reset Postgres before each DB profile so it sees the same clean, # freshly-seeded server a standalone `benchmark.sh ` run gets. # Postgres is started once and shared across the whole run, so otherwise the @@ -240,6 +244,17 @@ run_one() { postgres_start fi PG_DIRTY=true + if db_split_fits; then + GCANNON_CPUS="$DB_LOADGEN_CPUS" + THREADS="$DB_THREADS" + postgres_pin "$DB_POSTGRES_CPUS" + info "cpus: server $PROF_CPU | generator $GCANNON_CPUS ($THREADS threads) | postgres $DB_POSTGRES_CPUS" + fi + ;; + gateway-64|gateway-h3|production-stack) + # A database profile leaves it pinned, on what are this profile's + # generator CPUs. + postgres_pin ;; esac diff --git a/scripts/check_badge_parity.js b/scripts/check_badge_parity.js index 1084d7ee86..fd52dd445c 100644 --- a/scripts/check_badge_parity.js +++ b/scripts/check_badge_parity.js @@ -99,7 +99,7 @@ const run = new Function('D', ` `)(D); // ── rank the same way write_badges() does ─────────────────────────────────── -const FAMILIES = ['h1', 'h2', 'h3', 'gw', 'grpc', 'ws']; +const FAMILIES = ['h1', 'h2', 'h3', 'gw', 'grpc', 'ws', 'db']; const LEAGUES = [['flagship', 'emerging'], ['engine'], ['experimental'], ['infrastructure']]; const MIN_FIELD = 2; diff --git a/scripts/gen_leaderboard_data.py b/scripts/gen_leaderboard_data.py index c7a11011b1..157c58e673 100644 --- a/scripts/gen_leaderboard_data.py +++ b/scripts/gen_leaderboard_data.py @@ -103,12 +103,12 @@ ("static-tls","Static TLS", "20-file static serving over TLS (reference for frameworks).", [1024], [1024], False,False,True), ]), ("Database", [ - # Reference-only. The database and its driver dominate these two far - # more than the framework does, which is the case #1310 makes: they - # measure the connector, not the HTTP path. Still measured and shown, - # just no longer deciding the ranking. - ("async-db", "Async DB", "Async Postgres sequential scan (reference).", [1024], [1024], False,True,False), - ("fortunes", "Fortunes", "DB query + HTML template render (reference).", [1024], [1024], False,False,False), + # Their own family. The database and its driver dominate these two far + # more than the framework does, which is the case #1310 makes: that rules + # them out of a ranking of HTTP paths, and it is what this one ranks. They + # run over HTTP/1.1 with the box split three ways, see DB_* in profiles.sh. + ("async-db", "Async DB", "Async Postgres sequential scan.", [1024], [1024], True,True,False), + ("fortunes", "Fortunes", "DB query + HTML template render.", [1024], [1024], True,False,False), ]), ("HTTP/2", [ ("baseline-h2", "Baseline", "Baseline over h2 (TLS, ALPN).", [256,1024], [256,1024], True,True,True), @@ -167,8 +167,8 @@ "latency-1m": "test-profiles/h1/latency-1m/implementation", "latency-10k": "test-profiles/h1/latency-10k/implementation", "latency-500k-8cpu": "test-profiles/h1/latency-500k-8cpu/implementation", - "async-db": "test-profiles/h1/async-database/implementation", - "fortunes": "test-profiles/h1/fortunes/implementation", + "async-db": "test-profiles/database/async-database/implementation", + "fortunes": "test-profiles/database/fortunes/implementation", "baseline-h2": "test-profiles/h2/baseline-h2/implementation", "static-h2": "test-profiles/h2/static-h2/implementation", "baseline-h2c": "test-profiles/h2/baseline-h2c/implementation", @@ -1250,11 +1250,11 @@ def _slug(name): return (re.sub(r"[^A-Za-z0-9._-]+", "-", name).strip("-") or "unnamed").lower() # category -> family key. Mirrors familyOf() in index.html; anything unlisted is -# HTTP/1.1 (Connection, Workload, Database, Multi-endpoint). +# HTTP/1.1 (Connection, Concurrency, Efficiency, Workload). FAMILY_OF_CAT = {"Gateway": "gw", "HTTP/2": "h2", "HTTP/3": "h3", - "gRPC": "grpc", "WebSocket": "ws"} + "gRPC": "grpc", "WebSocket": "ws", "Database": "db"} FAMILY_LABEL = {"h1": "H/1.1", "h2": "H/2", "h3": "H/3", - "gw": "Gateway", "grpc": "gRPC", "ws": "WebSocket"} + "gw": "Gateway", "grpc": "gRPC", "ws": "WebSocket", "db": "Database"} # Leagues are the board's type filter. flagship+emerging is its default view and # ranks as one field; engine and infrastructure entries are each scored on their @@ -1846,16 +1846,18 @@ def compute_achievements(agg, profiles, meta, badge_index): OG_MUTED = (154, 160, 166) SCOPE_NAME = {"h1": "HTTP/1.1", "h2": "HTTP/2", "h3": "HTTP/3", - "gw": "Gateway", "grpc": "gRPC", "ws": "WebSocket"} + "gw": "Gateway", "grpc": "gRPC", "ws": "WebSocket", "db": "Database"} # The board's

per family. renderHead() writes the same strings, so the # pre-rendered heading and the rendered one match. SCOPE_H1 = {"h1": "HTTP/1.1 web server benchmarks", "h2": "HTTP/2 web server benchmarks", "h3": "HTTP/3 web server benchmarks", "gw": "Gateway and production stack benchmarks", - "grpc": "gRPC server benchmarks", "ws": "WebSocket server benchmarks"} + "grpc": "gRPC server benchmarks", "ws": "WebSocket server benchmarks", + "db": "Database-backed web server benchmarks"} SCOPE_BLURB = {"h1": "all HTTP/1.1 profiles", "h2": "all HTTP/2 profiles", "h3": "all HTTP/3 profiles", "gw": "the gateway & production-stack profiles", - "grpc": "all gRPC profiles", "ws": "all WebSocket profiles"} + "grpc": "all gRPC profiles", "ws": "all WebSocket profiles", + "db": "the database profiles"} # The board's own default: view=composite, scope=h1, flagship+emerging, tuned # shown, useMem and rescale off. Prerendering anything else would show a crawler # a page no visitor lands on. @@ -2760,7 +2762,8 @@ def _lang_faq(lang, scopes, n_entries, round_name): continue fw, _k, _t, score, rank, of, _l, _lr, _lo = rows[0] qa.append(( - f"What is the fastest {lang} {SCOPE_NAME[scope]} server?", + f"What is the fastest {lang} " + f"{'database-backed' if scope == 'db' else SCOPE_NAME[scope]} server?", f"{fw}, with a composite of {score:.0f} on {SCOPE_NAME[scope]} and " f"{rank} of {of} against all languages in its league.")) qa.append(( diff --git a/scripts/lib/postgres.sh b/scripts/lib/postgres.sh index 3e59b02f69..5081355afa 100644 --- a/scripts/lib/postgres.sh +++ b/scripts/lib/postgres.sh @@ -56,6 +56,14 @@ postgres_start() { fail "postgres sidecar did not become ready within 60s" } +# Move the running sidecar onto these CPUs. With no argument it goes back to where +# postgres_start() puts it: PG_CPUSET if set, otherwise anywhere. +postgres_pin() { + local cpus="${1:-${PG_CPUSET:-0-$(( $(nproc) - 1 ))}}" + docker update --cpuset-cpus="$cpus" "$PG_CONTAINER" >/dev/null \ + || warn "could not pin postgres to cpuset=$cpus" +} + postgres_stop() { # -v for the same reason as postgres_start: nuke any attached # anonymous volumes along with the container. diff --git a/scripts/lib/profiles.sh b/scripts/lib/profiles.sh index 2707f2bcf0..dcc27a85ae 100644 --- a/scripts/lib/profiles.sh +++ b/scripts/lib/profiles.sh @@ -9,6 +9,17 @@ # # Adding a profile: add a line to PROFILES and append to PROFILE_ORDER. +# The database profiles split the box three ways instead of two, because Postgres +# does real work on them and needs CPUs of its own: 32 logical CPUs for the server, +# 32 for the load generator and the rest for Postgres. Each is a run of cores plus +# their SMT siblings (N and N+64 on the bench host), the way every other profile +# pairs them. The other profiles give the server and the generator 64 each, which +# leaves Postgres nowhere to run but on top of them. +DB_SERVER_CPUS="${DB_SERVER_CPUS:-0-15,64-79}" +DB_LOADGEN_CPUS="${DB_LOADGEN_CPUS:-16-31,80-95}" +DB_POSTGRES_CPUS="${DB_POSTGRES_CPUS:-32-63,96-127}" +DB_THREADS="${DB_THREADS:-32}" + declare -A PROFILES=( [baseline]="1|0|0-31,64-95|4096|" [pipelined]="16|0|0-31,64-95|4096|pipeline" @@ -92,8 +103,8 @@ declare -A PROFILES=( # reporting a lower number that reads like a like-for-like result. [8gbit]="1|0|0-31,64-95|512|8gbit" [static-tls]="1|200|0-31,64-95|1024|static-tls" - [async-db]="1|0|0-31,64-95|1024|async-db" - [fortunes]="1|0|0-31,64-95|1024|fortunes" + [async-db]="1|0|$DB_SERVER_CPUS|1024|async-db" + [fortunes]="1|0|$DB_SERVER_CPUS|1024|fortunes" [baseline-h2]="1|0|0-31,64-95|256,1024|h2" [static-h2]="1|0|0-31,64-95|256,1024|static-h2" [baseline-h2c]="1|0|0-31,64-95|256,1024,4096|h2c" @@ -144,6 +155,15 @@ parse_profile() { IFS='|' read -r PROF_PIPELINE PROF_REQ PROF_CPU PROF_CONNS PROF_ENDPOINT <<< "$spec" } +# Does every CPU the database split names exist on this box? On one that is too +# small, run_one() leaves the generator and Postgres where they are, the same way +# framework_start() falls back when a profile's cpuset does not fit. +db_split_fits() { + local highest + highest=$(grep -oE '[0-9]+' <<< "$DB_SERVER_CPUS,$DB_LOADGEN_CPUS,$DB_POSTGRES_CPUS" | sort -n | tail -1) + [ "$highest" -lt "$(nproc 2>/dev/null || echo 1)" ] +} + # Map an endpoint to the tool name that handles it. # Returns one of: gcannon, wrk, zrk, h2load, h2load-h3 endpoint_tool() { diff --git a/scripts/validate.sh b/scripts/validate.sh index f10af4413c..3c4f85ce13 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -2180,7 +2180,7 @@ fi # ───── Async Database (GET /async-db) ───── if has_test "async-db"; then - ASYNCDB_DOCS="$DOCS_BASE/h1/async-database/validation" + ASYNCDB_DOCS="$DOCS_BASE/database/async-database/validation" echo "[test] async-db endpoint" asyncdb_fail=false # ranges and limits drawn per run @@ -2242,7 +2242,7 @@ fi # content, and is sized like a real page (not stripped to win the bench). if has_test "fortunes"; then - FORTUNES_DOCS="$DOCS_BASE/h1/fortunes/validation" + FORTUNES_DOCS="$DOCS_BASE/database/fortunes/validation" echo "[test] fortunes endpoint" body=$(curl -s --max-time 30 "http://localhost:$PORT/fortunes" || true) diff --git a/site/content/docs/add-framework/badges.md b/site/content/docs/add-framework/badges.md index 094477b46f..9d4dd68648 100644 --- a/site/content/docs/add-framework/badges.md +++ b/site/content/docs/add-framework/badges.md @@ -26,12 +26,13 @@ under `scopes..default.markdown`. | Family | `` | What the composite covers | |---|---|---| -| HTTP/1.1 | `h1` | Connection, Workload, Database and Multi-endpoint profiles | +| HTTP/1.1 | `h1` | Connection, Workload and Multi-endpoint profiles | | HTTP/2 | `h2` | the HTTP/2 profiles | | HTTP/3 | `h3` | the HTTP/3 profiles | | Gateway | `gw` | reverse-proxy and production-stack profiles | | gRPC | `grpc` | unary and streaming, plaintext and TLS | | WebSocket | `ws` | the echo profiles | +| Database | `db` | the Postgres-backed profiles | `` is your entry's board name, lowercased, with anything outside `A-Z a-z 0-9 . _ -` replaced by `-`. It is the same name as your file under diff --git a/site/content/docs/running-locally/configuration.md b/site/content/docs/running-locally/configuration.md index 18659a87b4..68861b7557 100644 --- a/site/content/docs/running-locally/configuration.md +++ b/site/content/docs/running-locally/configuration.md @@ -40,6 +40,10 @@ Every framework `Dockerfile` reads the same defaults from its env, so you rarely | `LOADGEN_DOCKER` | `false` | When `true`, every load generator runs from its Docker image instead of the host binary. Builds missing images automatically from `docker/*.Dockerfile`. Forced `true` by `benchmark-lite.sh`. | | `GCANNON_MODE` | `native` | `native` or `docker`. Implied by `LOADGEN_DOCKER`. | | `GCANNON_CPUS` | dynamic | CPU list the load generators run on. Native mode wraps calls in `taskset -c $GCANNON_CPUS`; docker mode passes it via `--cpuset-cpus`. Defaults to the upper half of available cores on smaller machines to avoid resource contention with the server. | +| `DB_SERVER_CPUS` | `0-15,64-79` | Server cpuset on the database profiles. | +| `DB_LOADGEN_CPUS` | `16-31,80-95` | Load-generator cpuset on the database profiles. Replaces `GCANNON_CPUS` there. | +| `DB_POSTGRES_CPUS` | `32-63,96-127` | Postgres cpuset on the database profiles. On a box too small for these three, the generator and Postgres stay where they are. | +| `DB_THREADS` | `32` | Load-generator threads on the database profiles. Replaces `THREADS` there. | ## Tool binaries and images diff --git a/site/content/docs/running-locally/scripts/benchmark.md b/site/content/docs/running-locally/scripts/benchmark.md index f0e96ced63..b8e27a7ea0 100644 --- a/site/content/docs/running-locally/scripts/benchmark.md +++ b/site/content/docs/running-locally/scripts/benchmark.md @@ -91,6 +91,10 @@ LOADGEN_DOCKER=true ./scripts/benchmark.sh aspnet-minimal | `LOADGEN_DOCKER` | `false` | When `true`, every load generator runs from its Docker image instead of the host binary. Builds missing images automatically. | | `GCANNON_MODE` | `native` | `native` or `docker`. `LOADGEN_DOCKER=true` sets this to `docker` for you. | | `GCANNON_CPUS` | `32-63,96-127` | Cores the load generators are pinned to (via `taskset` in native mode or `--cpuset-cpus` in docker mode). Framework containers use the *other* half of the CPU topology. | +| `DB_SERVER_CPUS` | `0-15,64-79` | Server cpuset on the database profiles. | +| `DB_LOADGEN_CPUS` | `16-31,80-95` | Load-generator cpuset on the database profiles. Replaces `GCANNON_CPUS` there. | +| `DB_POSTGRES_CPUS` | `32-63,96-127` | Postgres cpuset on the database profiles. On a box too small for these three, the generator and Postgres stay where they are. | +| `DB_THREADS` | `32` | Load-generator threads on the database profiles. Replaces `THREADS` there. | ### Tool binaries + images @@ -131,7 +135,7 @@ pipeline | req_per_conn | cpu_limit | connections | endpoint | `api-4` | 1 | 5 | `0-3` | 256 | gcannon | 8-template mix (baseline / json / async-db) | | `api-16` | 1 | 5 | `0-7,64-71` | 1024 | gcannon | 8-template mix | | `static` | 1 | 200 | `0-31,64-95` | 1024, 4096, 6800 | wrk | 20 files via `static-rotate.lua` | -| `async-db` | 1 | ∞ | `0-31,64-95` | 1024 | gcannon | 5 limit variants (5/10/20/35/50), `-r 25` | +| `async-db` | 1 | ∞ | `0-15,64-79` | 1024 | gcannon | 5 limit variants (5/10/20/35/50), `-r 25` | | `baseline-h2` | 1 | ∞ | `0-31,64-95` | 256, 1024 | h2load | `/baseline2` on `H2PORT` TLS | | `static-h2` | 1 | ∞ | `0-31,64-95` | 256, 1024 | h2load | `/static/*` on `H2PORT` TLS | | `baseline-h3` | 1 | ∞ | `0-31,64-95` | 64 | h2load-h3 | `/baseline2` on `H2PORT` QUIC | diff --git a/site/content/docs/scoring/achievements.md b/site/content/docs/scoring/achievements.md index 1a9512268d..d0cde5806b 100644 --- a/site/content/docs/scoring/achievements.md +++ b/site/content/docs/scoring/achievements.md @@ -21,7 +21,7 @@ Nothing is scored again to award them. A medal is a place in a field the site al Every field is the entry's own league. Frameworks, engines and reverse proxies are three separate competitions, and a framework never places against an engine. -A profile only awards a medal to an entry that is scored on it. Reference-only profiles, Pipelined and Fortunes, award nothing to frameworks, and an engine holds no medal on a profile outside the engine set. +A profile only awards a medal to an entry that is scored on it. Reference-only profiles such as Pipelined award nothing to frameworks, and an engine holds no medal on a profile outside the engine set. Tuned entries follow the badge rule: a standard entry is placed in the field without tuned entries, a tuned entry in the field with them. It is always the smallest published field the entry belongs to. diff --git a/site/content/docs/scoring/composite-score.md b/site/content/docs/scoring/composite-score.md index 4c147e9ea8..197802e7d3 100644 --- a/site/content/docs/scoring/composite-score.md +++ b/site/content/docs/scoring/composite-score.md @@ -64,8 +64,6 @@ Not all profiles count toward the composite score. Profiles marked as **scored** | JSON Compressed | Yes | JSON with `Accept-Encoding: gzip, br` and multiplier `?m=N` | | JSON TLS | Yes | JSON workload over HTTP/1.1 + TLS on port 8081 | | Async Delay | Yes | `GET /delay/10` over 32,000 held connections - what the framework does while a request waits. Scored for frameworks and engines, not for infrastructure | -| Async DB | No (*) | Async Postgres query with connection pooling. Reference-only since #1331 - the driver dominates the result more than the framework does | -| Fortunes | No (*) | DB query + HTML template render. Reference-only - engine-comparison test, not part of the composite ranking | ### H/1.1 Workload @@ -115,9 +113,16 @@ Not all profiles count toward the composite score. Profiles marked as **scored** | Echo Pipelined | Yes | Batched WebSocket echo throughput | | Echo Short-lived | Yes | WebSocket echo with each connection closed after 10 messages | -Fortunes, Pipelined, Static TLS and Async DB are the reference-only profiles - shown on the board as faded columns for comparison, but not counted in the composite score. +### Database -The two database profiles were scored until recently. They stopped because the database and its driver dominate them far more than the framework does: a framework's `async-db` number mostly reports which Postgres driver its language has, which is not what this board sets out to compare. They are still run and still published, because the number is worth having; it just no longer decides the ranking. +| Profile | Scored | Workload | +|---|---|---| +| Async DB | Yes | Async Postgres query with connection pooling | +| Fortunes | Yes for frameworks, not for engines | DB query + HTML template render | + +These two rank on their own board and count toward no other composite. The database and its driver dominate them far more than the framework does: a framework's `async-db` number mostly reports which Postgres driver its language has, which rules it out of a ranking of HTTP paths and is what this board ranks instead. + +Pipelined and Static TLS are the reference-only profiles - shown on the board as faded columns for comparison, but not counted in the composite score. ## Memory efficiency bonus @@ -171,7 +176,7 @@ B actually wins the memory term despite A's 5× throughput advantage, because `s Types are scored **separately** - each has its own composite ranking and normalization pool. The scored profiles differ by type: -- **Frameworks** (Flagship, Emerging and Experimental, in either Standard or Tuned mode) are scored on all scored profiles across H/1.1, H/2, H/3, gRPC, and WebSocket. +- **Frameworks** (Flagship, Emerging and Experimental, in either Standard or Tuned mode) are scored on all scored profiles across H/1.1, H/2, H/3, gRPC, WebSocket and Database. - **Engines** are scored on a reduced set: Baseline, Short-lived, JSON TLS, async-db, H/2 (all three), H/3 (both), gRPC (all four), the gateway profiles, and WebSocket, since most engines don't implement the heavier endpoints (compression negotiation, the TLS echo). - **Infrastructure** (nginx, Caddy, h2o and similar proxies / static-file servers) is scored on the nine profiles a server can answer without an application framework behind it: Baseline, Pipelined, Short-lived, JSON TLS, Static TLS, and Baseline and Static over both H/2 and H/3. Upload, the database profiles, gRPC, WebSocket and the gateway stacks may be displayed as reference data but do not count. diff --git a/site/content/docs/test-profiles/_index.md b/site/content/docs/test-profiles/_index.md index 374cc9c423..dafd9c2b9b 100644 --- a/site/content/docs/test-profiles/_index.md +++ b/site/content/docs/test-profiles/_index.md @@ -30,7 +30,7 @@ Five load generators are dispatched per profile - each one is built for a specif | Parameter | Value | |-----------|-------| | Load generators | `gcannon` (HTTP/1.1, WebSocket), `wrk` (static, json-tls and 8gbit rotation), `h2load` (HTTP/2, h2c, gateway), `h2load-h3` (HTTP/3 / QUIC), `zrk` (fixed-rate) | -| Threads | 64 for `gcannon` / `wrk` / `h2load` / `h2load-h3` (`$THREADS` / `$H2THREADS` / `$H3THREADS`) | +| Threads | 64 for `gcannon` / `wrk` / `h2load` / `h2load-h3` (`$THREADS` / `$H2THREADS` / `$H3THREADS`); 32 on the [database profiles](database/) (`$DB_THREADS`) | | Duration | 5s default; `async-db` 10s (hardcoded in the profile dispatcher) | | Runs | 3 per (profile, connection count) - best RPS wins | | Networking | Docker `--network host` for all containers (server + load generator + Postgres + Redis sidecars) | @@ -63,4 +63,5 @@ Gateway and `production-stack` profiles are compose-orchestrated, so their servi {{< card link="h3" title="H/3" subtitle="Baseline and static file benchmarks over QUIC for frameworks with native H/3 support." icon="globe-alt" >}} {{< card link="grpc" title="gRPC" subtitle="Unary RPC throughput over cleartext HTTP/2 using Protocol Buffers serialization." icon="globe-alt" >}} {{< card link="ws" title="WebSocket" subtitle="WebSocket echo throughput measuring frame processing performance." icon="globe-alt" >}} + {{< card link="database" title="Database" subtitle="Postgres-backed HTTP/1.1 benchmarks with the server, load generator and database each on CPUs of their own." icon="database" >}} {{< /cards >}} diff --git a/site/content/docs/test-profiles/database/_index.md b/site/content/docs/test-profiles/database/_index.md new file mode 100644 index 0000000000..ec23beee71 --- /dev/null +++ b/site/content/docs/test-profiles/database/_index.md @@ -0,0 +1,23 @@ +--- +weight: 6 +title: Database +seo_title: "Database Test Profiles" +description: "Postgres-backed HTTP/1.1 benchmarks that rank the whole path from request to query and back, with the server, load generator and database each on CPUs of their own." +--- + +Database profiles measure the whole path from request to query and back: the framework, its Postgres driver and any data-access layer on top. They run over HTTP/1.1 with the same load generator as the H/1.1 profiles, but rank on their own board, because the database and its driver dominate these numbers far more than the framework does. + +## CPU layout + +Postgres does real work here, so the box is split three ways instead of two. Every core is paired with its SMT sibling. + +| Role | Logical CPUs | Cpuset | +|---|---:|---| +| Server | 32 | `0-15,64-79` | +| Load generator (32 threads) | 32 | `16-31,80-95` | +| Postgres | 64 | `32-63,96-127` | + +{{< cards >}} + {{< card link="async-database" title="Async Database (Postgres)" subtitle="Async Postgres range query over 100K rows, connection pooling, and JSON serialization." icon="database" >}} + {{< card link="fortunes" title="Fortunes (Templates)" subtitle="DB query + HTML template rendering with auto-escape." icon="document-text" >}} +{{< /cards >}} diff --git a/site/content/docs/test-profiles/h1/async-database/_index.md b/site/content/docs/test-profiles/database/async-database/_index.md similarity index 100% rename from site/content/docs/test-profiles/h1/async-database/_index.md rename to site/content/docs/test-profiles/database/async-database/_index.md diff --git a/site/content/docs/test-profiles/h1/async-database/implementation.md b/site/content/docs/test-profiles/database/async-database/implementation.md similarity index 93% rename from site/content/docs/test-profiles/h1/async-database/implementation.md rename to site/content/docs/test-profiles/database/async-database/implementation.md index d936332df8..f466d90284 100644 --- a/site/content/docs/test-profiles/h1/async-database/implementation.md +++ b/site/content/docs/test-profiles/database/async-database/implementation.md @@ -10,15 +10,15 @@ The Async Database profile measures how efficiently a framework handles concurre **This test is for framework-type entries only** - engines (nginx, h2o, etc.) are excluded. -> **Reference-only.** This profile is measured and published but does not contribute to the -> composite score. The database and its driver dominate the result far more than the framework -> does, so the number mostly reports which Postgres driver a language has rather than anything -> about the HTTP path this board sets out to compare. It is still worth having, it just no -> longer decides the ranking. +> **Scored on the Database board only.** The database and its driver dominate the result far +> more than the framework does, so the number mostly reports which Postgres driver a language +> has. That rules it out of the HTTP/1.1 composite, and it is what the Database board ranks. **Connections:** 1,024 +**CPUs:** 32 for the server, 32 for the load generator and the rest for Postgres - see the [CPU layout](../#cpu-layout). + ## How it works 1. A Postgres container runs alongside the framework container on the same host, listening on `localhost:5432` diff --git a/site/content/docs/test-profiles/h1/async-database/validation.md b/site/content/docs/test-profiles/database/async-database/validation.md similarity index 100% rename from site/content/docs/test-profiles/h1/async-database/validation.md rename to site/content/docs/test-profiles/database/async-database/validation.md diff --git a/site/content/docs/test-profiles/h1/fortunes/_index.md b/site/content/docs/test-profiles/database/fortunes/_index.md similarity index 88% rename from site/content/docs/test-profiles/h1/fortunes/_index.md rename to site/content/docs/test-profiles/database/fortunes/_index.md index 512adf4fa0..acae3d3ba0 100644 --- a/site/content/docs/test-profiles/h1/fortunes/_index.md +++ b/site/content/docs/test-profiles/database/fortunes/_index.md @@ -4,7 +4,7 @@ seo_title: "Fortunes Template Rendering Benchmark" description: "Database-backed HTML rendering end to end: a Postgres read, runtime row injection, sorting and escape-aware template output." --- -Database-backed HTML rendering benchmark - measures template-engine throughput end-to-end: Postgres read, runtime row injection, sort, escape-aware HTML rendering. Reference-only profile (not part of the composite score). +Database-backed HTML rendering benchmark - measures template-engine throughput end-to-end: Postgres read, runtime row injection, sort, escape-aware HTML rendering. Scored on the Database board. {{< cards >}} {{< card link="implementation" title="Implementation Guidelines" subtitle="Endpoint specification, expected output, and type-specific rules." icon="code" >}} diff --git a/site/content/docs/test-profiles/h1/fortunes/implementation.md b/site/content/docs/test-profiles/database/fortunes/implementation.md similarity index 95% rename from site/content/docs/test-profiles/h1/fortunes/implementation.md rename to site/content/docs/test-profiles/database/fortunes/implementation.md index 3a12bbfbc2..f433cb6e23 100644 --- a/site/content/docs/test-profiles/h1/fortunes/implementation.md +++ b/site/content/docs/test-profiles/database/fortunes/implementation.md @@ -9,10 +9,12 @@ The Fortunes profile measures template-engine throughput on a realistic page-ren **Diverges from TechEmpower Fortunes (12-row spec) by design.** With only 12 rows the per-request render time is dwarfed by the PG round-trip, so the profile ends up measuring the database driver more than the template engine. Bumping the seed to 200 rows scales render cost linearly while leaving query cost effectively constant (still one round-trip, still single-page table read), shifting the bottleneck from PG-throughput to framework-CPU - which is what the profile name actually claims to measure. -**This test is reference-only - it does not contribute to the composite score.** It exists for engine-to-engine comparison (Razor vs Jinja2 vs ERB vs Askama, etc.), not framework ranking. Frameworks without a template engine should leave it out of `meta.json.tests`. +**This test is scored on the Database board for frameworks, not for engines, and is not part of the HTTP/1.1 composite.** Frameworks without a template engine should leave it out of `meta.json.tests`, and score 0 for it on that board. **Connections:** 1,024 +**CPUs:** 32 for the server, 32 for the load generator and the rest for Postgres - see the [CPU layout](../#cpu-layout). + ## How it works 1. A Postgres container runs alongside the framework container on the same host, listening on `localhost:5432`. The same connection pool used by `async-db` is shared. diff --git a/site/content/docs/test-profiles/h1/fortunes/validation.md b/site/content/docs/test-profiles/database/fortunes/validation.md similarity index 100% rename from site/content/docs/test-profiles/h1/fortunes/validation.md rename to site/content/docs/test-profiles/database/fortunes/validation.md diff --git a/site/content/docs/test-profiles/gateway/gateway-h2/implementation.md b/site/content/docs/test-profiles/gateway/gateway-h2/implementation.md index 89f147789c..82ace3686a 100644 --- a/site/content/docs/test-profiles/gateway/gateway-h2/implementation.md +++ b/site/content/docs/test-profiles/gateway/gateway-h2/implementation.md @@ -42,7 +42,7 @@ See the individual test profile docs for each endpoint's exact request/response - [`/static/*`](/docs/test-profiles/h1/static-tls/implementation/) - [`/baseline2`](/docs/test-profiles/h2/baseline-h2/implementation/) - [`/json`](/docs/test-profiles/h1/json-tls/implementation/) -- [`/async-db`](/docs/test-profiles/h1/async-database/implementation/) +- [`/async-db`](/docs/test-profiles/database/async-database/implementation/) ## Docker Compose diff --git a/site/content/docs/test-profiles/gateway/gateway-h2/validation.md b/site/content/docs/test-profiles/gateway/gateway-h2/validation.md index 9d85726a82..a3e600cee5 100644 --- a/site/content/docs/test-profiles/gateway/gateway-h2/validation.md +++ b/site/content/docs/test-profiles/gateway/gateway-h2/validation.md @@ -66,7 +66,7 @@ Sends `GET /async-db?min=10&max=50` over HTTP/2 and validates: - Every item has a `tags` array - Every item's `active` field is a boolean (not a string or integer) -This is the same validation as the [Async Database test](../../h1/async-database/validation), but routed through the proxy. +This is the same validation as the [Async Database test](../../database/async-database/validation), but routed through the proxy. ### Content-Type header @@ -92,7 +92,7 @@ The gateway-64 test relies on endpoint implementations that are validated indivi - `/json` - [JSON Compressed validation](/docs/test-profiles/h1/json-compressed/validation) - `/static/*` - [Static Files validation](/docs/test-profiles/h1/static-tls/validation) -- `/async-db` - [Async Database validation](/docs/test-profiles/h1/async-database/validation) +- `/async-db` - [Async Database validation](/docs/test-profiles/database/async-database/validation) - `/baseline2` - [Baseline H2 validation](/docs/test-profiles/h2/baseline-h2/validation) Frameworks subscribing to `gateway-64` should also subscribe to these individual tests to ensure each endpoint passes validation independently before testing through the proxy. diff --git a/site/content/docs/test-profiles/h1/_index.md b/site/content/docs/test-profiles/h1/_index.md index d40d5a11da..ec0770deeb 100644 --- a/site/content/docs/test-profiles/h1/_index.md +++ b/site/content/docs/test-profiles/h1/_index.md @@ -2,7 +2,7 @@ weight: 1 title: H/1.1 seo_title: "HTTP/1.1 Test Profiles" -description: "Single-endpoint HTTP/1.1 benchmarks, each isolating one dimension: throughput, JSON, TLS echo, static files, concurrency, CPU cost and database access." +description: "Single-endpoint HTTP/1.1 benchmarks, each isolating one dimension: throughput, JSON, TLS echo, static files, concurrency and CPU cost." --- Every HTTP/1.1 profile measures one thing at a time, so a number can be attributed to a single part of the framework rather than to a mix. @@ -17,8 +17,6 @@ Every HTTP/1.1 profile measures one thing at a time, so a number can be attribut {{< card link="latency-10k" title="Latency-10K" subtitle="Latency-1M's setup at a near-idle 10K req/s; the metric is the CPU and latency of a server with almost nothing to do." icon="chip" >}} {{< card link="latency-1m" title="Latency-1M" subtitle="One million req/s offered at a fixed rate; the metric is the CPU each framework spends to serve it, read exactly from its cgroup." icon="chip" >}} {{< card link="latency-500k-8cpu" title="Latency-500K/8 *" subtitle="500K req/s pinned with the server on four cores plus their SMT siblings - near saturation, so the queue a server builds shows in its mean latency. Reference-only while the first entries are measured." icon="chip" >}} - {{< card link="async-database" title="Async Database (Postgres)" subtitle="Async Postgres range query over 100K rows, connection pooling, and JSON serialization. Framework-only benchmark." icon="database" >}} {{< card link="static-tls" title="Static Files over TLS" subtitle="Same 20-file static workload transported over HTTP/1.1 + TLS on port 8081 - measures the cost of encryption on bulk file serving." icon="lock-closed" >}} {{< card link="pipelined" title="Pipelined (16x) *" subtitle="16 requests sent back-to-back per connection, testing raw I/O and pipeline batching. Reference-only - not part of the composite score." icon="fast-forward" >}} - {{< card link="fortunes" title="Fortunes (Templates) *" subtitle="DB query + HTML template rendering with auto-escape. Reference-only - measures template-engine throughput, not part of the composite score." icon="document-text" >}} {{< /cards >}} diff --git a/site/leaderboard/index.html b/site/leaderboard/index.html index 945155a24d..48db3c176e 100644 --- a/site/leaderboard/index.html +++ b/site/leaderboard/index.html @@ -744,13 +744,13 @@

Object.keys(D.results).forEach(function(k){ D.results[k].forEach(function(r){ if(r.lang) FWLANG[r.fw]=r.lang; }); }); var COMPOSITE = '__composite__'; - var SCOPES = [{k:'h1',label:'H/1.1'},{k:'h2',label:'H/2'},{k:'h3',label:'H/3'},{k:'gw',label:'Gateway'},{k:'grpc',label:'gRPC'},{k:'ws',label:'WebSocket'}]; - function familyOf(p){ return {'Gateway':'gw','HTTP/2':'h2','HTTP/3':'h3','gRPC':'grpc','WebSocket':'ws'}[p.category]||'h1'; } + var SCOPES = [{k:'h1',label:'H/1.1'},{k:'h2',label:'H/2'},{k:'h3',label:'H/3'},{k:'gw',label:'Gateway'},{k:'grpc',label:'gRPC'},{k:'ws',label:'WebSocket'},{k:'db',label:'Database'}]; + function familyOf(p){ return {'Gateway':'gw','HTTP/2':'h2','HTTP/3':'h3','gRPC':'grpc','WebSocket':'ws','Database':'db'}[p.category]||'h1'; } // cats must list every CATALOG category that belongs to the family. This is // the nav's only source of profiles, so a category missing here disappears // from Detailed Views entirely while still ranking on the composite -- // which is how async and latency-1m went missing when they were added. - var FAMILIES=[{k:'h1',label:'HTTP/1.1',cats:['Connection','Concurrency','Efficiency','Workload','Database']},{k:'h2',label:'HTTP/2',cats:['HTTP/2']},{k:'h3',label:'HTTP/3',cats:['HTTP/3']},{k:'gw',label:'Gateway',cats:['Gateway']},{k:'grpc',label:'gRPC',cats:['gRPC']},{k:'ws',label:'WebSocket',cats:['WebSocket']}]; + var FAMILIES=[{k:'h1',label:'HTTP/1.1',cats:['Connection','Concurrency','Efficiency','Workload']},{k:'h2',label:'HTTP/2',cats:['HTTP/2']},{k:'h3',label:'HTTP/3',cats:['HTTP/3']},{k:'gw',label:'Gateway',cats:['Gateway']},{k:'grpc',label:'gRPC',cats:['gRPC']},{k:'ws',label:'WebSocket',cats:['WebSocket']},{k:'db',label:'Database',cats:['Database']}]; var expanded=new Set(['composite']); // state.lang is an exact language filter, distinct from the q search box: q // matches substrings across name/language/engine, which is fine for typing but @@ -1389,10 +1389,10 @@

var cat=document.getElementById('pcat'), title=document.getElementById('ptitle'), blurb=document.getElementById('pblurb'), ctl=document.getElementById('headctl'); if(state.view==='composite'){ - var sLabel={h1:'all HTTP/1.1 profiles',h2:'all HTTP/2 profiles',h3:'all HTTP/3 profiles',gw:'the gateway & production-stack profiles',grpc:'all gRPC profiles',ws:'all WebSocket profiles'}[state.scope]; + var sLabel={h1:'all HTTP/1.1 profiles',h2:'all HTTP/2 profiles',h3:'all HTTP/3 profiles',gw:'the gateway & production-stack profiles',grpc:'all gRPC profiles',ws:'all WebSocket profiles',db:'the database profiles'}[state.scope]; // The heading is the page's

. "HTTP/1.1" alone said nothing about // what is on the page, so every view names its subject. - var sName={h1:'HTTP/1.1 web server benchmarks',h2:'HTTP/2 web server benchmarks',h3:'HTTP/3 web server benchmarks',gw:'Gateway and production stack benchmarks',grpc:'gRPC server benchmarks',ws:'WebSocket server benchmarks'}[state.scope]; + var sName={h1:'HTTP/1.1 web server benchmarks',h2:'HTTP/2 web server benchmarks',h3:'HTTP/3 web server benchmarks',gw:'Gateway and production stack benchmarks',grpc:'gRPC server benchmarks',ws:'WebSocket server benchmarks',db:'Database-backed web server benchmarks'}[state.scope]; cat.textContent='Composite ranking'; title.textContent=sName; blurb.innerHTML='Normalized score summed across '+sLabel+' (each profile worth 100 to its leader' +(state.rescale?' among the frameworks shown':' in the full field, so filtering does not change the numbers') From 4333f2583a518bf0a585bea68a97556301d5a053 Mon Sep 17 00:00:00 2001 From: MDA2AV Date: Thu, 5 Nov 2026 23:25:03 +0000 Subject: [PATCH 2/2] benchmark: only async-db counts on the Database board for now Fortunes goes back to reference-only, shown as a faded column. 63 of the 101 entries running async-db have no Fortunes, so scoring it would cap all of them at half the board. --- README.md | 4 ++-- scripts/gen_leaderboard_data.py | 4 +++- site/content/docs/add-framework/badges.md | 2 +- site/content/docs/scoring/achievements.md | 2 +- site/content/docs/scoring/composite-score.md | 6 +++--- site/content/docs/test-profiles/database/_index.md | 4 ++-- site/content/docs/test-profiles/database/fortunes/_index.md | 2 +- .../docs/test-profiles/database/fortunes/implementation.md | 2 +- 8 files changed, 14 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index c2bd1e229e..b1c91c999a 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ The reply states which baseline it used, and profiles the other framework does n | Concurrency | `async` | `GET /delay/{ms}` with the wait named in the route, 32K held connections. Isolates what a framework does while a request is pending, with no database or I/O in the way. Scored for frameworks and engines, not for infrastructure, since the spec settled at a flat 10 ms over 32K connections | | Efficiency | `latency-1m`, `latency-10k`, `latency-500k-8cpu` * | One million req/s offered at a fixed rate with zrk; the metric is CPU spent per request, read exactly from the container's cgroup rather than sampled. Everyone who finishes serves the same million, so what separates them is the bill. Scored on its own 0-100 formula rather than on rps, since the rate is pinned. `latency-10k` is the identical setup - same cores, connections and endpoint - at a near-idle 10K req/s, so what it reports is the standing cost of being a running server rather than of serving load, and it is scored the same way. Both score CPU at 0.50, p99 at 0.25 and mean latency at 0.25, the mean being the queueing signal: ten times the best mean is worth nothing. `latency-500k-8cpu` pins 500K req/s with the server on four cores plus their SMT siblings, so the load sits near saturation and the queue a server builds shows in its mean; reference-only while the first entries are measured | | Workload | `json-comp`, `json-tls`, `8gbit`, `static-tls` | JSON serialization, gzip/brotli compression, HTTP/1.1 over TLS, a 10 KB body echoed over TLS at a pinned 50K req/s so ingest and egress load together - scored on the same 0-100 formula as the latency profiles, since its rate is pinned too, 20-file static asset serving over TLS | -| Database | `async-db`, `fortunes` | Postgres-backed HTTP/1.1, on their own board: an async range query, and a DB query + HTML template render (TechEmpower-style Fortunes). The server gets 32 logical CPUs, the load generator 32 and Postgres the rest | +| Database | `async-db`, `fortunes` * | Postgres-backed HTTP/1.1, on their own board: an async range query, and a DB query + HTML template render (TechEmpower-style Fortunes, reference-only). The server gets 32 logical CPUs, the load generator 32 and Postgres the rest | | H/2 | `baseline-h2`, `static-h2`, `baseline-h2c`, `json-h2c` | Baseline + static over TLS with h2 stream multiplexing; baseline + JSON over cleartext h2 (prior-knowledge, port 8082) | | H/3 | `baseline-h3`, `static-h3` | Baseline and static over QUIC with TLS 1.3 | | gRPC | `unary-grpc`, `unary-grpc-tls` | Unary gRPC over plaintext HTTP/2 and TLS | @@ -104,7 +104,7 @@ Swap `actix` for your entry and `h1` for the family you want: | Gateway | `gw` | reverse-proxy and production-stack profiles | | gRPC | `grpc` | unary and streaming, plaintext and TLS | | WebSocket | `ws` | the echo profiles | -| Database | `db` | the Postgres-backed profiles | +| Database | `db` | the Async DB profile | `` is your entry's name on the board, lowercased, with anything outside `A-Z a-z 0-9 . _ -` turned into `-`, the same name as your file in diff --git a/scripts/gen_leaderboard_data.py b/scripts/gen_leaderboard_data.py index 157c58e673..fffdfae19d 100644 --- a/scripts/gen_leaderboard_data.py +++ b/scripts/gen_leaderboard_data.py @@ -107,8 +107,10 @@ # more than the framework does, which is the case #1310 makes: that rules # them out of a ranking of HTTP paths, and it is what this one ranks. They # run over HTTP/1.1 with the box split three ways, see DB_* in profiles.sh. + # Only async-db counts for now: 63 of the 101 entries running it have no + # Fortunes, and scoring it would cap all of them at half the board. ("async-db", "Async DB", "Async Postgres sequential scan.", [1024], [1024], True,True,False), - ("fortunes", "Fortunes", "DB query + HTML template render.", [1024], [1024], True,False,False), + ("fortunes", "Fortunes", "DB query + HTML template render (reference).", [1024], [1024], False,False,False), ]), ("HTTP/2", [ ("baseline-h2", "Baseline", "Baseline over h2 (TLS, ALPN).", [256,1024], [256,1024], True,True,True), diff --git a/site/content/docs/add-framework/badges.md b/site/content/docs/add-framework/badges.md index 9d4dd68648..f1c506ac51 100644 --- a/site/content/docs/add-framework/badges.md +++ b/site/content/docs/add-framework/badges.md @@ -32,7 +32,7 @@ under `scopes..default.markdown`. | Gateway | `gw` | reverse-proxy and production-stack profiles | | gRPC | `grpc` | unary and streaming, plaintext and TLS | | WebSocket | `ws` | the echo profiles | -| Database | `db` | the Postgres-backed profiles | +| Database | `db` | the Async DB profile | `` is your entry's board name, lowercased, with anything outside `A-Z a-z 0-9 . _ -` replaced by `-`. It is the same name as your file under diff --git a/site/content/docs/scoring/achievements.md b/site/content/docs/scoring/achievements.md index d0cde5806b..1a9512268d 100644 --- a/site/content/docs/scoring/achievements.md +++ b/site/content/docs/scoring/achievements.md @@ -21,7 +21,7 @@ Nothing is scored again to award them. A medal is a place in a field the site al Every field is the entry's own league. Frameworks, engines and reverse proxies are three separate competitions, and a framework never places against an engine. -A profile only awards a medal to an entry that is scored on it. Reference-only profiles such as Pipelined award nothing to frameworks, and an engine holds no medal on a profile outside the engine set. +A profile only awards a medal to an entry that is scored on it. Reference-only profiles, Pipelined and Fortunes, award nothing to frameworks, and an engine holds no medal on a profile outside the engine set. Tuned entries follow the badge rule: a standard entry is placed in the field without tuned entries, a tuned entry in the field with them. It is always the smallest published field the entry belongs to. diff --git a/site/content/docs/scoring/composite-score.md b/site/content/docs/scoring/composite-score.md index 197802e7d3..a4c9b4b046 100644 --- a/site/content/docs/scoring/composite-score.md +++ b/site/content/docs/scoring/composite-score.md @@ -118,11 +118,11 @@ Not all profiles count toward the composite score. Profiles marked as **scored** | Profile | Scored | Workload | |---|---|---| | Async DB | Yes | Async Postgres query with connection pooling | -| Fortunes | Yes for frameworks, not for engines | DB query + HTML template render | +| Fortunes | No (*) | DB query + HTML template render. Reference-only - engine-comparison test, not part of the composite ranking | -These two rank on their own board and count toward no other composite. The database and its driver dominate them far more than the framework does: a framework's `async-db` number mostly reports which Postgres driver its language has, which rules it out of a ranking of HTTP paths and is what this board ranks instead. +Async DB ranks on its own board and counts toward no other composite. The database and its driver dominate it far more than the framework does: a framework's `async-db` number mostly reports which Postgres driver its language has, which rules it out of a ranking of HTTP paths and is what this board ranks instead. -Pipelined and Static TLS are the reference-only profiles - shown on the board as faded columns for comparison, but not counted in the composite score. +Fortunes, Pipelined and Static TLS are the reference-only profiles - shown on the board as faded columns for comparison, but not counted in the composite score. ## Memory efficiency bonus diff --git a/site/content/docs/test-profiles/database/_index.md b/site/content/docs/test-profiles/database/_index.md index ec23beee71..80f74036f8 100644 --- a/site/content/docs/test-profiles/database/_index.md +++ b/site/content/docs/test-profiles/database/_index.md @@ -5,7 +5,7 @@ seo_title: "Database Test Profiles" description: "Postgres-backed HTTP/1.1 benchmarks that rank the whole path from request to query and back, with the server, load generator and database each on CPUs of their own." --- -Database profiles measure the whole path from request to query and back: the framework, its Postgres driver and any data-access layer on top. They run over HTTP/1.1 with the same load generator as the H/1.1 profiles, but rank on their own board, because the database and its driver dominate these numbers far more than the framework does. +Database profiles measure the whole path from request to query and back: the framework, its Postgres driver and any data-access layer on top. They run over HTTP/1.1 with the same load generator as the H/1.1 profiles, but rank on their own board, because the database and its driver dominate these numbers far more than the framework does. Async Database is the profile that counts; Fortunes is shown for reference. ## CPU layout @@ -19,5 +19,5 @@ Postgres does real work here, so the box is split three ways instead of two. Eve {{< cards >}} {{< card link="async-database" title="Async Database (Postgres)" subtitle="Async Postgres range query over 100K rows, connection pooling, and JSON serialization." icon="database" >}} - {{< card link="fortunes" title="Fortunes (Templates)" subtitle="DB query + HTML template rendering with auto-escape." icon="document-text" >}} + {{< card link="fortunes" title="Fortunes (Templates) *" subtitle="DB query + HTML template rendering with auto-escape. Reference-only - measures template-engine throughput, not part of the composite score." icon="document-text" >}} {{< /cards >}} diff --git a/site/content/docs/test-profiles/database/fortunes/_index.md b/site/content/docs/test-profiles/database/fortunes/_index.md index acae3d3ba0..512adf4fa0 100644 --- a/site/content/docs/test-profiles/database/fortunes/_index.md +++ b/site/content/docs/test-profiles/database/fortunes/_index.md @@ -4,7 +4,7 @@ seo_title: "Fortunes Template Rendering Benchmark" description: "Database-backed HTML rendering end to end: a Postgres read, runtime row injection, sorting and escape-aware template output." --- -Database-backed HTML rendering benchmark - measures template-engine throughput end-to-end: Postgres read, runtime row injection, sort, escape-aware HTML rendering. Scored on the Database board. +Database-backed HTML rendering benchmark - measures template-engine throughput end-to-end: Postgres read, runtime row injection, sort, escape-aware HTML rendering. Reference-only profile (not part of the composite score). {{< cards >}} {{< card link="implementation" title="Implementation Guidelines" subtitle="Endpoint specification, expected output, and type-specific rules." icon="code" >}} diff --git a/site/content/docs/test-profiles/database/fortunes/implementation.md b/site/content/docs/test-profiles/database/fortunes/implementation.md index f433cb6e23..667cca7349 100644 --- a/site/content/docs/test-profiles/database/fortunes/implementation.md +++ b/site/content/docs/test-profiles/database/fortunes/implementation.md @@ -9,7 +9,7 @@ The Fortunes profile measures template-engine throughput on a realistic page-ren **Diverges from TechEmpower Fortunes (12-row spec) by design.** With only 12 rows the per-request render time is dwarfed by the PG round-trip, so the profile ends up measuring the database driver more than the template engine. Bumping the seed to 200 rows scales render cost linearly while leaving query cost effectively constant (still one round-trip, still single-page table read), shifting the bottleneck from PG-throughput to framework-CPU - which is what the profile name actually claims to measure. -**This test is scored on the Database board for frameworks, not for engines, and is not part of the HTTP/1.1 composite.** Frameworks without a template engine should leave it out of `meta.json.tests`, and score 0 for it on that board. +**This test is reference-only - it does not contribute to the composite score.** It exists for engine-to-engine comparison (Razor vs Jinja2 vs ERB vs Askama, etc.), not framework ranking. Frameworks without a template engine should leave it out of `meta.json.tests`. **Connections:** 1,024