[zijiecode]

No description provided.

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[1am9trash]

/reuse-sweep-run 40714803265

[1am9trash]

[1am9trash]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40714803265
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40714803265
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Cookbook PR: sgl-project/sglang#47394

AMD monokernel eval:
The TP8 c13 eval server log shows monokernel muse_spark_dflash2_draft_hip on gfx950 runs target_verify steps of batch size [1, 2] x 8 tokens (2 of 2 captured decode graphs) on every rank, including TP7/EP7. This is SGLang’s post-capture routing check, confirming that the monokernel served the eval.

TP4 HiCache production settings:
The prefill sizes, memory fractions, allocator setting, intervals, and HiCache sizes are per-concurrency selections on the published TEP4 recipe. The cookbook documents the shared baseline; these measured MI355X values are not a separate serving configuration.

Throughput parity:
The removed TP8/EP8 c64 point is 32849.8 tok/s/GPU, 0.8% above TP4 c40 at 32588.5. That gap is within normal run-to-run measurement noise, while TP4/EP4 raises interactivity from 76.3 to 116.3 tok/s. The TEP4 curve therefore covers the removed point.

Draft precision:
RadixArk/Muse-Spark-Open-DFlash2 is bfloat16 and has no quantization_config. The launch sets no draft quantization, dtype, or KV override, and SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is absent. The FP8 KV cache applies to both target and draft.

Signed: @1am9trash

[github-actions]

❌❌❌ REJECTED ❌❌❌

@1am9trash: three blocking issues on 2803f36. (1) The merged cookbook does not cover several production settings of the TP4/EP4 HiCache variant. (2) The removed TP8 c64 point is still on the frontier, and no head point covers it. (3) No log this verifier could read shows the AMD monokernel path running during the eval.

❌ Check 2 (Evals pass): FAIL — Eval coverage of the conditional AMD monokernel path could not be verified. The TP8 c13 eval job log (job 122910588567) shows only the --monokernel-backend amd launch flag. The muse_spark_dflash2_draft_hip ... target_verify routing line quoted in the sign-off would be in the eval_server_logs_musespark_tp8_conc13_... tarball, which this verifier's sandbox could not extract. A maintainer must confirm that line in the artifact. Accuracy itself passes on the 1027 image. gsm8k em_strict is 0.9773 (TP8 c13), 0.9765 (TP4 c24) and 0.9780 (TP4 c40). All three are within 2σ of their in-run siblings and of the exact ingested cohort (run 39179279826, run 39659301076).

❌ Check 3 (Upstream recipe): FAIL — The merged cookbook leaves out production settings of the TP4/EP4 HiCache variant (c10–c40). The cookbook is sglang#47394, merged 2026-10-30T05:24Z, MuseSparkOpen.mdx. Its "HiCache (high concurrency)" block does not cover these settings: --chunked-prefill-size/--max-prefill-tokens 8192/16384; --mem-fraction-static 0.86–0.96 with SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1 (c28–c40); PYTORCH_HIP_ALLOC_CONF=expandable_segments:True (every TP4 point); --cuda-graph-max-bs-decode 4–72; --prefill-decode-interval 24–96 (the cookbook base is 20); and --hicache-size 253 at c10–c24 (the cookbook gives only 310). The allocator setting and the constant chunked-prefill sizes are fixed serving settings, not per-concurrency tuning. The cookbook gives no tuning rule from which a user could derive the others. The additional detail section does not argue that any of these settings is InferenceX-only.

❌ Check 18 (Recipe/image/sweep consistency): FAIL — (g) The removed point musespark tp8 ep8 conc64 is still undominated, and the head curve does not cover it. It measured 32849.8 tok/s/GPU at 76.3 tok/s interactivity on the 1020 image (run 39659301076). The highest head throughput is TP4 c40 at 32588.5. Its interactivity of 76.3 is below the lowest non-dominated head point (c40 at 116.3), so no head segment brackets it and extrapolation is not allowed. The sign-off's "within run-to-run noise" argument is not one of the allowed coverage conditions. Keep a head point that covers it, or add one. (a)–(f) pass: both search spaces generate; the master conc lists, TP/EP and gpus: 4 overrides match the recipe; the image resolves by digest; and the changelog is accurate. (h) Re-tune deltas: c1 +19.9% tput / +49.0% interactivity, c4 +12.0% / +43.1%, c8 +2.7% / +52.1%.

⚠️ Check 13 (Draft precision): WARN — Draft precision could not be verified for musespark-fp4-mi355x-sglang-agentic-dflash (all points), so human review is needed. What was verified: RadixArk/Muse-Spark-Open-DFlash2 config.json is bfloat16 with no quantization_config. The pinned recipe sets no draft quantization, dtype or draft-KV override; the FP8 KV cache applies to target and draft alike. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set in the recipe or the logged environment. What is missing: source or build provenance for how the muse-spark-open-1027 model-enablement image loads the DFlash2 draft by default.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @1am9trash is a listed owner of inferencex-e2e/configs/amd-master.yaml and srt-slurm-recipes/**/mi355x-*/. perf-changelog.yaml falls under the catch-all.

✅ Check 1 (In-PR sweep + evals): PASS — In-PR commit c999c25 has 10/10 agentic / and 3/3 agentic eval / check-runs with conclusion success (run 40714803265). The later merge 2803f36 keeps the recipe blob (1ea4e7f) and the master entry unchanged. Its base changes are the DSV4-Pro/M3 retirement and the srt-slurm ae3d687→59920ea bump. They touch only other models and Dynamo-sidecar, opt-in GPU-index, container-reuse, multi-node topology and sa-bench code, none of which this single-node SGLang agg path reaches.

✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 40714803265 was posted on its own line by 1am9trash (COLLABORATOR).

✅ Check 5 (Checklist template): PASS — Every item of the current template is present and checked, including the upstream-recipe sub-item.

✅ Check 6 (Upstream image): PASS — lmsysorg/sglang-rocm:muse-spark-open-1027@sha256:7184d3ae… is from the upstream lmsysorg org. An SGLang-only entry is not subject to the ordering rule. The image is a model-enablement build for a new model (musespark, added 2026-10-15), which the main-branch policy temporarily allows.

✅ Check 7 (Deprecated models): PASS — Muse Spark Open agentic coding is listed as active in MODELS.md as of 2026-10-30.

✅ Check 8 (Architecture hacks): PASS — There are no hf-overrides, model-config edits or indexer/skip knobs. musespark is in the policy-baseline matrix (2a45791), so it is grandfathered for integer all-reduce.

✅ Check 9 (Chat template): PASS — The AgentX client replays chat-format coding traces (semianalysis_cc_traces_weka_062126), and acceptance is simulated per Check 11.

✅ Check 10 (No engine patches): PASS — The recipe and its launcher path reach no patches, setup scripts, source edits or wheel installs.

✅ Check 11 (Golden AL): PASS — The benchmark logs show SGLANG_SIMULATE_ACC_LEN=4.05 with match-expected/real-draft-token. This matches musespark_dflash.yaml thinking_on at 7 draft tokens (speculative-num-draft-tokens 8).

➖ Check 12 (Append-only): N/A — No new changelog entry sets append-only: true.

✅ Check 14 (Pareto coverage): PASS — There is one curve: musespark / agentic-coding / MI355X SGLang / fp4 / run 40714803265 (1027 image), p90 E2EL vs total tput/GPU, from the bmk_agentic_* artifacts. 7 of its 10 measured points are on the frontier: c4, c8, c10, c16, c24, c28 and c40.

➖ Check 15 (PR description): N/A — The PR body is empty.

➖ Check 16 (NVIDIA overview): N/A — Only MI355X (AMD) configurations are affected.

➖ Check 17 (Metrics publication): N/A — This is a direct SGLang deployment, not Dynamo TRT-LLM.

Assessed commit: 2803f36ac4e09aedaf5d18b84d436c9365acb4c3.