Repository navigation
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
65ddfae to
606eac8
Compare
xinli-sw
added
the
non-canary-full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| args: | ||
| tensor-parallel-size: 4 | ||
| max-num-seqs: 96 | ||
| kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":964000000000,"lazy_offload":true}}' |
There was a problem hiding this comment.
🔴 The two TP4 points (conc 64 and 96) pin almost all of a GB300 node's job memory as KV pool, so they can be OOM-killed at startup or under load. cpu_bytes_to_use 964000000000 is 919,346 MiB. The exclusive job gets 920,000 MiB, so about 650 MiB is left for every vLLM process and the client. Fix: keep a real margin for the TP4 pool, e.g. dram-utilization near 0.85-0.9 with cpu_bytes_to_use to match. Other full-node pools use 0.80-0.95. Same pattern at 2 sites (agg-variants.yaml:72, agg-variants.yaml:97); master:8901 sets the 1.0.
Why this was flagged
The TP4 master rows at inferencex-e2e/configs/nvidia-master.yaml:8904 and :8906 sit under dram-utilization: 1.0 (:8901) on cluster:gb300-nv. They select override_tp4_c64 and override_tp4_c96, which set cpu_bytes_to_use to 964000000000 (agg-variants.yaml:72, :97). runners.yaml:641 gives gb300-nv available-cpu-dram-mib: 920_000, which is 964,689,920,000 bytes. A single-node job is exclusive by default (single_node_exclusive default True, infx/clusters/slurm.py:191), so it requests --mem=0 and its whole budget is those 920,000 MiB. The pinned pool leaves about 658 MiB for the vLLM API server, the engine core, the 4 TP workers (CUDA contexts, torch), the AgentX client and the container. On base, the TP4 high-concurrency point was the DEP4 row with kv-offloading: none, which pins no host pool. The binder check at infx/srt_slurm/single_node.py:144-146 compares only metadata, not headroom, so nothing catches this before submission.
Verification: agg-variants.yaml:72 (override_tp4_c64) and :97 (override_tp4_c96) both set "cpu_bytes_to_use":964000000000. That is about 919,342 MiB. runners.yaml:641 gives gb300-nv available-cpu-dram-mib: 920_000, which is 964,689,920,000 bytes. So the KV pool is about 99.93% of the node's job-usable DRAM, leaving roughly 660 MiB.
| kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":482000000000,"lazy_offload":true}}' | ||
| benchmark: | ||
| env: | ||
| CONC: '48' |
There was a problem hiding this comment.
🟡 nit (optional): If dram-utilization or the runner's DRAM metadata changes, the run reports a DRAM budget that differs from what vLLM pins, and nothing catches it. Each override hard-codes cpu_bytes_to_use (482 GB at TP2, 964 GB at TP4) but leaves out TOTAL_CPU_DRAM_GB. That key is what single_node.py:144 checks against the matrix. Fix: add TOTAL_CPU_DRAM_GB: '482' to the two TP2 overrides and '964' to the two TP4 overrides under benchmark.env, as the B200 sibling recipe does. The binder then fails on any drift.
Why this was flagged
In inferencex-e2e/infx/srt_slurm/single_node.py:144-146, the binder checks CONC, KV_OFFLOADING and TOTAL_CPU_DRAM_GB against the matrix point, but only for keys the recipe's benchmark env declares. The four new overrides (agg-variants.yaml:53-100) declare only CONC. The matrix computes TOTAL_CPU_DRAM_GB from available-cpu-dram-mib: 920_000 (runners.yaml gb300-nv) × dram-utilization: 1.0 × gpus/4, giving 964/482 today, so the values match for now. infx/results/agentic/init.py:192 records that matrix value as allocated_cpu_dram_gb. If either input changes, the reported allocation and the bytes vLLM actually pins split silently. The B200 recipe (b200-fp4-mtp/agentic.yaml, override_tp4_c32_dram) avoids this by declaring TOTAL_CPU_DRAM_GB: '1073'.
Verification: The trigger is a later edit to the gb300-nv available-cpu-dram-mib value or to this entry's dram-utilization. Nothing is wrong today: the values match. inferencex-e2e/infx/srt_slurm/single_node.py:144-146 checks TOTAL_CPU_DRAM_GB only if the recipe declares it, and the four new overrides declare only CONC. If either input changes, infx/results/agentic/__init__.py:192 records the wrong allocated_cpu_dram_gb and no check flags it.
| - { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 32], srt-recipe: agentic.yaml } | ||
| - { tp: 4, ep: 4, dp-attn: true, spec-decoding: mtp, kv-offloading: none, conc-list: [64, 96, 128], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml } | ||
| - dram-utilization: 1.0 |
There was a problem hiding this comment.
🟡 nit (optional): Maintainers are left with a dead recipe variant. The removed DEP4 row was the only user of zip_override_dep4 in gb300-fp4-mtp/agentic.yaml. AGENTS.md:74 says to delete recipes that no active config uses. Fix: delete the zip_override_dep4 block from minimaxm31flash/vllm/gb300-fp4-mtp/agentic.yaml in this PR. Keep zip_override_tp4, which the unchanged TP4 conc 1-32 row still selects.
Why this was flagged
The diff removes the { tp: 4, ep: 4, dp-attn: true, ... conc-list: [64, 96, 128], router: vllm-router, srt-recipe: agentic.yaml } row from minimaxm31flash-fp4-gb300-vllm-agentic-dspark in nvidia-master.yaml (old line ~8905). That row was the only matrix point that could match zip_override_dep4 in inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/minimaxm31flash/vllm/gb300-fp4-mtp/agentic.yaml, which sets tp1/dp4/EP, the vllm-router frontend and CONC 64/96/128. A repo-wide grep shows nvidia-master.yaml:8896 is the only reference to minimaxm31flash/vllm/gb300-fp4-mtp. The remaining agentic.yaml row is TP4 with no EP, and validate_recipe in infx/srt_slurm/single_node.py rejects the DEP4 variant for that row. So after merge the block is unreachable dead config, which AGENTS.md:74 says to delete. Nothing fails at runtime, so this is a nit.
Verification: The diff deletes the only DEP4 row from minimaxm31flash-fp4-gb300-vllm-agentic-dspark (nvidia-master.yaml, around line 8905). zip_override_dep4 stays at minimaxm31flash/vllm/gb300-fp4-mtp/agentic.yaml:67. select_recipe (single_node.py:95-107) drops the dep4 variants on validation, so no matrix point can ever select the dep4 block any more.
606eac8 to
5958b9a
Compare
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40627456192 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40627456192 |
使用最新配置提交 MiniMax M3.1 AgentX。
5958b9a to
8461aab
Compare
…00-configs-local # Conflicts: # inferencex-e2e/perf-changelog.yaml
使用最新配置提交 MiniMax M3.1 AgentX。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
使用最新配置提交 MiniMax M3.1 AgentX。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # inferencex-e2e/perf-changelog.yaml
# Conflicts: # inferencex-e2e/perf-changelog.yaml
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
@xinli-sw staged run 40445022376.
This run remains available across future |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…00-dynagg # Conflicts: # inferencex-e2e/perf-changelog.yaml
xinli-sw
added
non-canary-full-sweep-enabled
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/use 40627456192 |
|
@xinli-sw staged run 40627456192. This run remains available across future |
Use latest configs for MiniMax M3.1 GB300 AgentX submission.
中文
更新 MiniMax M3.1 GB300 AgentX 提交配置。
AI model disclosure
Prepared by Codex with delegated agents (exact model/version identifiers could not be verified from the runtime) and Claude Opus 5.5 (claude-opus-5-5).
🤖 Generated with Claude Code