Repository navigation
Conversation
Adds a TP2 config key with L2 HiCache at conc 32 to 48; conc 44 to 48 use longest-prefix-match scheduling. 为 MiniMax-M3.1-Flash B300 SGLang AgentX 新增 TP2 + L2 HiCache 配置，覆盖 conc 32 到 48，其中 conc 44 到 48 使用 longest-prefix-match 调度。
ispobock
added
the
full-sweep-fail-fast
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # the decode graph batch matches it. | ||
|
|
||
| zip_override_tp2_hicache: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 2 | ||
| moe-runner-backend: flashinfer_trtllm_routed | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: [64, 72, 80] | ||
| cuda-graph-max-bs-decode: [64, 72, 80] | ||
| mem-fraction-static: 0.87 | ||
| enable-hierarchical-cache: true | ||
| hicache-size: 295 | ||
| hicache-write-policy: write_back | ||
| benchmark: | ||
| env: | ||
| CONC: ['32', '36', '40'] | ||
| KV_OFFLOADING: dram | ||
|
|
There was a problem hiding this comment.
🟡 nit (optional): New override and search-space comments are section headers that restate code, which AGENTS.md forbids. sweep:^# TP2 with L2 HiCache|^# Higher-concurrency points|# TP2 with L2 HiCache: 295 GB Examples: agentic-tp2.yaml:72 repeats schedule-policy: lpm, and nvidia-master.yaml:9484 repeats the tp, hicache size and conc-list values. Fix: delete these header comments. Keep only a one-line why at a specific line where needed, such as next to max-running-requests for the 2x CONC choice. [also at: inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/minimaxm31flash/sglang/b300-fp4-mtp/agentic-tp2.yaml:72 - nit: Two new comments only restate the config next to them, which the repo's comment rules forbid. "Higher-concurrency points use longest-prefix-match scheduling." just repeats schedule-policy: lpm, and the master-config comment "TP2 with L2 HiCache: 295 GB per rank at conc 32-48." repeats the row below it.]
Why this was flagged
AGENTS.md "Comments (mandatory, every file)" bans "header comments describing a file, topology, config, recipe, override, or section" and says "never restate the code". agentic-tp2.yaml:52-53 sits above zip_override_tp2_hicache and repeats write_back and 295 GB, which are already in the override's hicache-write-policy and hicache-size lines. agentic-tp2.yaml:72 sits above zip_override_tp2_hicache_lpm and only restates schedule-policy: lpm. nvidia-master.yaml:9484 "# TP2 with L2 HiCache: 295 GB per rank at conc 32-48." restates the search-space row below it. The sibling agentic.yaml has similar section comments, which is precedent but not a sanctioned exception. Nothing breaks at runtime; this is a convention slip that reviewers following AGENTS.md would flag.
Verification: AGENTS.md lines 54-58 say: "No header comments describing a file, topology, config, recipe, override, or section" and "never restate the code". agentic-tp2.yaml:51-52 and :72 are section headers that repeat hicache-size: 295 and schedule-policy: lpm. nvidia-master.yaml:9484 restates the row below it. Nothing functional breaks.
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#46506 has been merged. This PR's AgentX config ( This PR has no authorized |
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40634380871 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40634380871 |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#46547 has been merged. This PR's AgentX config ( This PR has no authorized |
ispobock
added
full-sweep-fail-fast
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
ispobock
added
full-sweep-fail-fast
ispobock
removed
the
full-sweep-fail-fast
ispobock
added
the
full-sweep-fail-fast
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#46807 has been merged. This PR's AgentX config ( This PR has no authorized |
|
/use 40345805111 |
|
@ispobock staged run 40345805111. This run remains available across future |
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
As a PR reviewer and CODEOWNER, I have reviewed this and have Check recipe, flags. Ignore missing checklist |
❌❌❌ REJECTED ❌❌❌❗❗ WARNING ❗❗ NVIDIA official recipes overview: model/framework coverage is missing or unverified. See Check 16. @Oseltamivir Blocking: no eval has run on any commit in this PR. The sign-off also has no checklist items and no upstream recipe link. ❌ Check 1 (Sweep + evals on in-PR commit): FAIL — No passing sweep/eval was found on any commit in this PR. Run 40345805111 on in-PR commit ❌ Check 2 (Eval accuracy): FAIL — Evals were skipped. No eval exists for the new TP2 recipe ❌ Check 3 (Upstream recipe): FAIL — The sign-off's additional detail section has no link to a merged SGLang cookbook PR or a published cookbook page. Its body is only "Check recipe, flags. Ignore missing checklist". Coverage does exist: sgl-project/sglang#46807 (MERGED 2026-10-27) adds the B300 TP2 entry from this PR at ❌ Check 5 (Latest checklist template): FAIL — The sign-off has none of the items in the current PR_REVIEW_CHECKLIST.md template: no checkboxes, no additional detail section and no signature. "Ignore missing checklist" does not explain a missing item.
Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — Oseltamivir is a CODEOWNER ( ✅ Check 4 (Reuse command): PASS — ispobock (COLLABORATOR) posted ✅ Check 6 (Upstream image / engine-first): PASS — ✅ Check 7 (Deprecated models): PASS — MODELS.md lists ✅ Check 8 (No architecture hacks): PASS — No ✅ Check 9 (Spec-decode chat template): PASS — The AgentX trace replay goes through the shared AIPerf chat-completions client, which this PR does not change. Acceptance is governed by Check 11. ✅ Check 10 (No engine patches): PASS — ✅ Check 11 (Golden simulated AL): PASS — The run logs show ✅ Check 12 (Append-only): PASS — The single new changelog entry is append-only. The 9 base points stay unchanged and 6 TP2 points are added (no points removed or re-tuned). The new recipe file and the new ✅ Check 14 (Pareto coverage): PASS — The combined B300 SGLang FP4 AgentX curve at P90 E2EL has 7/5 frontier points: TP2 c32/36/40/44/48 from run 40345805111 plus same-image TP4 c16 and dram-c32 from run 40178345296. The new TP2 points alone give 5/5; c46 is dominated. ✅ Check 15 (PR description): PASS — The body matches the pinned code and run artifacts: TP2 at conc 32-48, 370 GB HiCache per rank within the 749 GB allocation ( ✅ Check 18 (Recipe/image/sweep consistency): PASS — The config generates without errors. Master TP2 and the conc list match the recipe's ➖ Check 17 (Metrics publication): N/A — Direct SGLang, with no Dynamo TRT-LLM serving path. Assessed commit: |
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
Set speculative-draft-kv-cache-dtype fp8_e4m3 on every point, matching the vLLM DSpark drafter. The bf16 draft KV took 21 GB of HBM and 71 GB of the HiCache host pool per rank at TP2. Drop append-only so the whole curve reruns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zijiexia
added
full-sweep-fail-fast
The draft-only speculative-draft-kv-cache-dtype override lowers draft KV below the pinned image's bf16 default, which the draft-precision rule in CONTRIBUTING.md forbids. Restore the append-only TP2 addition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
Reviewed and approved: store the DSpark draft KV in FP8 on every point and rerun the full B300 SGLang AgentX curve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sglang-agentx-tp2
zijiexia
added
full-sweep-fail-fast
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR's |
|
[by Claude Code] sgl-project/sglang#46948 has been merged. This PR's AgentX config ( This PR's |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR's |
|
[by Claude Code] sgl-project/sglang#47027 has been merged. This PR's AgentX config ( This PR's |
Add TP2 points with DRAM KV offload at conc 32 to 48 to the existing B300 SGLang AgentX curve. The host cache is 370 GB per rank, inside the 749 GB allocation of a two-GPU point at DRAM utilization 1.0; conc 44 to 48 use longest-prefix-match scheduling. The recipe lives in its own file, so existing points are unchanged.
中文
为现有的 B300 SGLang AgentX 曲线新增 conc 32 到 48 的 TP2 + DRAM KV offload 点。每个 rank 的 host cache 为 370 GB，在 DRAM utilization 1.0 下两卡点 749 GB 的分配额度以内；conc 44 到 48 使用 longest-prefix-match 调度。recipe 单独成文件，现有的点不受影响。
AI model disclosure
Prepared with Claude Code using
claude-opus-5-5. Reviewed by the human author before submission.