Skip to content

docs(qwen35): add 9B and 27B single-GH200 serving numbers - #12

Merged
xiaguan merged 3 commits into
pegainfer-project:mainfrom
FeathBow:feat/qwen35-9b-27b-serving
Aug 24, 2026
Merged

docs(qwen35): add 9B and 27B single-GH200 serving numbers#12
xiaguan merged 3 commits into
pegainfer-project:mainfrom
FeathBow:feat/qwen35-9b-27b-serving

Conversation

@FeathBow

Copy link
Copy Markdown
Contributor

Description

  • Adds Qwen3.5-9B Serving Load and Qwen3.5-27B Serving Load to /models/qwen35-4b/, measured on 1x GH200 120GB (aarch64, sm_90) during the openinfer 509/510 rerun campaign: full c/QPS ladders with TTFT/TPOT p50+p99, long-context and c=120 overload behavior, tool-call round-trips, and greedy-vs-HF plus per-size logits-gate summaries.
    • 9B: openinfer main ffb959c4, Python vllm bench serve client, random 1024/128, greedy, default seed. Saturation ~6.7 req/s / ~857 out tok/s.
    • 27B: ffb959c4. Plateau ~2.7 req/s / ~340 out tok/s at qps≥8; the load-time budget line is quoted and explained — 17.2 GB KV, 5.5 GB prefill scratch, 18.8 GB recurrent-state reserve (two ~147 MB states per decode slot across the 64-slot capacity).
  • Rewrites the Notes section for the family: per-request recurrent-state sizes (~49 MB at 9B / ~147 MB at 27B), the load-time reserve and the --max-batch capacity/KV trade-off, the decodable-vocab selection bound (248077 of the 248320 padded lm_head rows), and a corrected CUDA-Graph statement — the server always captures and rejects --cuda-graph=false, so the previous "disable for debugging" note no longer holds.
  • Links the overview table row: Qwen3.5-4B / 9B / 27B → /models/qwen35-4b/.
  • Validation: npm run build + static tests 12/12.

@FeathBow
FeathBow force-pushed the feat/qwen35-9b-27b-serving branch from 5241646 to 3b0bf60 Compare August 21, 2026 20:16
@FeathBow
FeathBow force-pushed the feat/qwen35-9b-27b-serving branch from 51fb630 to 92a6bf8 Compare August 21, 2026 21:03

@xiaguan xiaguan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The Qwen3.5 family coverage, serving-table arithmetic, memory-budget calculations, runtime notes, and existing route coverage are consistent. The full website test suite passes.

@xiaguan
xiaguan merged commit 2564029 into pegainfer-project:main Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants