Skip to content

perf(amd): cap GLM-5.2 FP4 MI355X SGLang TP8 c4 admission at 2 / perf(amd): 将 GLM-5.2 FP4 MI355X SGLang TP8 c4 的准入上限降到 2 #4630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
EricKing626 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

perf(amd): cap GLM-5.2 FP4 MI355X SGLang TP8 c4 admission at 2 / perf(amd): 将 GLM-5.2 FP4 MI355X SGLang TP8 c4 的准入上限降到 2 #4630

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b54115a
perf(amd): cap GLM-5.2 FP4 MI355X SGLang TP8 c4 admission at 2
EricKing626 Oct 28, 2026
9a4ff5f
Merge branch 'main' into amd/glm5.2-fp4-mi355x-c4-admission-cap
EricKing626 Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ...ex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.2/sglang/mi355x-fp4-mtp/agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ base:
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:'

# One variant per point. Admission is 2x CONC; the decode graph batch follows it
# up to 64. The TP4 HiCache tier is 180 GB per rank.
# up to 64. The TP4 HiCache tier is 180 GB per rank. TP8 c4 is the exception:
# its slowest decodes are the ones sharing a batch, so admission is capped
# below CONC to trade TTFT for p90 interactivity; the graph batch stays at
# 2x CONC.
Comment on lines +52 to +55

@claude claude Bot Oct 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): The TP8 c4 rationale was added to the section header comment rather than next to the value it explains, about 140 lines away. AGENTS.md forbids section header comments and allows only short comments on a specific line. Fix: move a one-line why comment onto max-running-requests: 2 in override_tp8_c4 (around line 193), and leave the header comment as it was.

Why this was flagged

AGENTS.md:56 forbids header comments that describe a section or override, and AGENTS.md:58 limits comments to a short technical note on a specific line or block. The diff extends the header above override_tp4_c4_hicache (agentic.yaml:51-55) with four lines about override_tp8_c4, whose max-running-requests: 2 is at about line 193. A reader of that line sees no reason for the value. The header now also describes an exception for a block elsewhere in the file.

Verification: The diff adds four lines to the section header comment at agentic.yaml:51-55. The value it explains, max-running-requests: 2 in override_tp8_c4, is at line 193 with no comment of its own. AGENTS.md:56 says "No header comments describing a file, topology, config, recipe, override, or section." It is a rule violation, not a runtime problem.

override_tp4_c4_hicache:
roles:
agg:
Expand Down Expand Up @@ -187,7 +190,7 @@ override_tp8_c4:
args:
tensor-parallel-size: 8
expert-parallel-size: 1
max-running-requests: 8
max-running-requests: 2
cuda-graph-max-bs-decode: 8
benchmark:
env:
Expand Down
10 changes: 10 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11499,6 +11499,16 @@
- "Preserve all eight concurrency points (1, 16, 32, 48, 192, 256, 384, 512), prefill/decode topology, DSpark settings, UMBP offload, MoRI transfer, and benchmark client image. The workload binder supplies the serving image to every selected recipe variant."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4571

- config-keys:
- glm5.2-fp4-mi355x-sglang-agentic-mtp
scenario-type:
- agentic-coding
description:
- "TP8 concurrency 4 drops max-running-requests from 8 to 2 and keeps cuda-graph-max-bs-decode at 8. No other point moves: TP8 concurrency 1 and 2 and every TP4 HiCache point keep admission at 2x CONC."
- "At this point the slowest decodes are the ones that share a decode batch, so admission above the in-flight tail only widens the batch that tail runs in. A max-running-requests sweep at TP8 concurrency 4 measured interactivity at p90 TPOT 222.3 to 255.0 output tokens/s/user and the slowest decile's mean concurrent decode partners 1.68 to 0.71, against TTFT p90 1.07 to 1.22 s and 3911 to 3754 tokens/s/GPU; requests over the cap wait for a slot."
- "The image, EAGLE shape (5 steps, 6 draft tokens), chunked-prefill-size, mem-fraction-static, HiCache tier and the DSA triton prefill and decode backends are unchanged."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4630
Comment on lines +11502 to +11510

@claude claude Bot Oct 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 BLOCKING: New inferencex-e2e/perf-changelog.yaml entries must be appended to the END of the file. This one now sits between #4571 and the later entries (#4554, #4464, #4468, ...) that came in when main was merged into this branch.
Why it matters: The file is read in chronological order, oldest at the top and newest at the bottom. An entry in the middle breaks that order and can be missed when result collection reads the file.
Fix: Move this entry, unchanged, from here to after the last entry (#4468) at the bottom of the file.

Comment on lines +11502 to +11510

@claude claude Bot Oct 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The changelog validator and PR sweep will reject this PR, because the new block sits mid-file instead of at the tail. It was inserted before the existing minimaxm31flash entries (PRs #4464 and #4468), which are the last entries on the base. Fix: restore the base file byte-for-byte, then append this one block after the final #4468 entry. The changelog conflict helper (prepare_perf_changelog_merge resolve-conflict) does this, and it keeps every main entry in its original order.

Why this was flagged

On base 98f0d6e the file is 11558 lines and ends with the minimaxm31flash GB200 (#4464) and GB300 (#4468) entries. The diff adds the glm5.2-fp4-mi355x-sglang-agentic-mtp block at line 11502, ahead of them. validate_perf_changelog.compare_entries compares base and head entries by position (validate_perf_changelog.py:129-137), so the first displaced minimaxm31flash slot now holds the new entry and fails with "entry N changed; existing entries are immutable". validate_raw_change (validate_perf_changelog.py:172) also fails because head_raw no longer starts with base_raw. run-sweep.yml builds its matrix from the appended tail entries, so the sweep would be blocked or would pick up the wrong config keys. AGENTS.md:88 requires appending only at the tail.

Verification: Trigger: merging this PR through the mandated infx.workflows.merge_with_reuse path. The diff hunk @@ -11499,6 +11499,16 @@ puts the new block (#4630) before the minimaxm31flash-fp4-b200-sglang-agentic-dspark entry. It is not at the tail. merge_with_reuse.py:688 calls canonicalize_changelog(pr, repo). canonicalize_changelog does not catch the error, so the merge tool aborts.


- config-keys:
- minimaxm31flash-fp4-b200-sglang-agentic-dspark
- minimaxm31flash-fp4-gb200-sglang-agentic-dspark
Expand Down
Loading