Repository navigation
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
31c06bb to
dda9782
Compare
YukioZzz
added
the
full-sweep-enabled
dda9782 to
cbfaaa9
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40645935622 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40645935622 |
cbfaaa9 to
cec28da
Compare
更新五个 Kimi-K3 PD 点位：吞吐点启用 BF16 SSM，移除 1P1D c56 和 1P3D c12，增加 FP32 SSM 的 1P1D c1 K7 monokernel 点。临时 kernel 补丁单独提交。
导入 vLLM #60683、#58743、#60808、#63031 的公开提交，保留原始 SHA 与 cherry-pick 来源。安装器使用运行镜像对应的源码和 ROCm SDK 重建 HIP 扩展，便于 nightly 收录后整体移除。
官方 nightly 已包含四个依赖 PR，移除临时补丁、安装器及构建入口，仅保留 recipe 配置更新。
eaeee53 to
0902c81
Compare
固定包含所需上游功能的 nightly 提交和镜像 digest，确保各节点及重复运行使用同一镜像。
YukioZzz
added
non-canary-full-sweep-enabled
Resolve perf-changelog with the repository merge helper.
|
/reuse-sweep-run 40301194088 |
billishyahao
left a comment
billishyahao
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40301194088
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40301194088
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Recipe: vllm-project/vllm#63726
Signed: billishyahao
|
/use 40301194088 |
|
@Oseltamivir staged run 40301194088. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:Signed: billishyahao |
❌❌❌ REJECTED ❌❌❌@Oseltamivir this sign-off cannot stand. The signer does not own ❌ Check 0 (CODEOWNER): FAIL — ❌ Check 1 (Sweep on in-PR commit): FAIL — The passing sweep predates a material change in ❌ Check 2 (Eval accuracy): FAIL —
❌ Check 18 (Recipe/image/sweep consistency): FAIL — (f1) the new Passed and not applicable checks✅ Check 3 (Upstream recipe): PASS — vllm-project/vllm#63726 MERGED 2026-10-27 and is published at recipes.vllm.ai/moonshotai/Kimi-K3 (sourced from ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — every current template item is present and checked. ✅ Check 6 (Upstream image/ordering): PASS — ✅ Check 7 (Deprecated models): PASS — Kimi-K3 agentic coding is active in MODELS.md. ✅ Check 8 (No architecture hacks): PASS — no hf-overrides or FLOP reduction; INT4 QuickReduce is covered by tool-call evals on every point. ✅ Check 9 (Chat templates): PASS — the AgentX replay drives the chat-completions endpoint through vllm-router. ✅ Check 10 (No engine patches): PASS — no reachable patching or setup scripts; the pre-existing Ionic verbs-provider mount is a host RDMA driver library, not engine code. ✅ Check 11 (Golden AL): PASS — the driver injects synthetic acceptance; logs show ➖ Check 12 (Append-only): N/A — the new changelog entry has no ✅ Check 15 (PR description): PASS — the body's topology table, image, SSM dtypes, DRAM budget and golden-AL claims match the pinned code and run logs. ➖ Check 16 (NVIDIA overview): N/A — MI355X only; no NVIDIA configurations are affected. ➖ Check 17 (Metrics publication): N/A — vLLM with vllm-router; no Dynamo TRT-LLM path. Assessed commit: |
YukioZzz
added
full-sweep-enabled
Run Kimi-K3 FP4 PD on MI355X with
vllm/vllm-openai-rocm:nightly-c50aca14416ef29f6ed0ed256302f30646726dcd, TP8/DCP8 workers, FP8 KV cache, and MoRIIO RDMA transfer. The image is pinned by digest. The recipe provides five configurations:The c8 decode worker uses
latency_specializedwith the V2 model runner and graph capture sizes[1, 8]. The four offload configurations use BF16 SSM state on both prefill and decode, with prefill-only SimpleCPUOffload at DRAM utilization 0.8 (2399 GB per prefill worker).DSpark uses
/models/Inferact-Kimi-K3-DSparkwith probabilistic drafting and block rejection; throughput runs use the selected golden acceptance curve, while evals use real verification.The recipe uses
vllm-routerwith consistent-hash routing and the standard eval selection. Recipe documentation: vllm-project/vllm#63726.AI model disclosure
AI assistance prepared the recipe and validation, reviewed upstream dependencies, and rebased the PR. A separate agent previously checked documentation discoverability. Exact model identifiers could not be verified from the runtime.
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
在 MI355X 上使用官方
vllm/vllm-openai-rocm:nightly-c50aca14416ef29f6ed0ed256302f30646726dcd运行 Kimi-K3 FP4 PD，P/D worker 均采用 TP8/DCP8、FP8 KV cache，通过 MoRIIO RDMA 传输 KV，镜像以 digest 固定。五种配置见上表：1P1D c8 使用 FP32 SSM、K7 DSpark，不启用 CPU offload；decode 端使用 V2 model runner 的
latency_specialized后端，graph capture sizes 为[1, 8]。其余四个吞吐配置使用 BF16 SSM、K4 DSpark 和仅 P 侧启用的 SimpleCPUOffload，DRAM utilization 为 0.8，每个 P worker 的预算为 2399 GB。DSpark draft 为
/models/Inferact-Kimi-K3-DSpark，使用 probabilistic drafting 和 block rejection。吞吐测试采用对应的 golden acceptance curve，eval 使用真实验证。路由采用vllm-routerconsistent hash，评测沿用标准选择。AI 协助配置、验证、依赖核对及 rebase；此前有独立 agent 检查文档可发现性。精确模型标识无法从运行环境核实。