Repository navigation
DeepSeek-V4-Pro-0813 B200 FP4 vLLM AgentX/ DeepSeek-V4-Pro-0813 B200 FP4 vLLM AgentX - #4554
hshrivastava-droid wants to merge 3 commits into
Conversation
…10/c16/c32 configs / DeepSeek-V4-Pro-0813 B200 FP4 vLLM AgentX：提交 ARIA 选中的 c1/c4/c10/c16/c32 配置 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
hshrivastava-droid
added
the
full-sweep-enabled
hshrivastava-droid
added
the
full-sweep-enabled
|
Thanks for the contribution!
中文感谢你的贡献！
|
hshrivastava-droid
added
full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40585790288 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40585790288 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🟡
inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv4/vllm/b200-fp4-mtp/agentic.yaml— No config runsoverride_tp8_c1oroverride_tp8_c4in agentic.yaml anymore, so they are dead variants. The base key's TP8 row is now conc [6, 14]. The lowconc key runs c1/c4 from agentic-lowconc.yaml. No other config points at this recipe dir. Fix: delete both blocks, as AGENTS.md requires for configs no active config uses.Why this was flagged
nvidia-master.yaml:905 narrows the dsv4-fp4-b200-vllm-agentic-mtp TP8 row to conc-list [6, 14]. nvidia-master.yaml:922 sends c1/c4 for the lowconc key to agentic-lowconc.yaml. The only references to dsv4/vllm/b200-fp4-mtp are nvidia-master.yaml:899 and :917. So no matrix point picks agentic.yaml:66
override_tp8_c1or agentic.yaml:82override_tp8_c4. Before this change the base key ran both. They are now stale copies that differ from the live agentic-lowconc.yaml values (53687091200 vs 42949672960 KV bytes, different cudagraph sizes). AGENTS.md "Deprecating benchmark configs" says to delete recipes no active config uses.Verification: nit. The base key's TP8 row narrows from
conc-list: [1, 4, 6, 10, 14, 16]to[6, 14], and the lowconc key sends c1/c4 to agentic-lowconc.yaml.validate_recipematchesbenchmark.env.CONC(infx/srt_slurm/single_node.py:145-147), sooverride_tp8_c1(agentic.yaml:66) andoverride_tp8_c4(agentic.yaml:82) are never selected. Nothing breaks.
| kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":120000000000,"enable_cross_layers_blocks":"true","lazy_offload":false}}' | ||
| tensor-parallel-size: 8 | ||
| data-parallel-size: 1 | ||
| max-num-batched-tokens: 16384 | ||
| kv-cache-memory-bytes: 53687091200 | ||
| max-num-seqs: 16 | ||
| compilation-config: '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,100,105,112,200,300,400,500]}' | ||
| env: | ||
| VLLM_DSV4_MEGA_FP8_COMBINE: null | ||
| benchmark: | ||
| env: | ||
| CONC: '16' | ||
| KV_OFFLOADING: none | ||
| KV_OFFLOADING: dram |
There was a problem hiding this comment.
🔴 The new c16 DRAM-offload result will record 1932 GB of allocated CPU DRAM, but the recipe pins only about 960 GB. The override sets cpu_bytes_to_use_per_rank to 120000000000 across 8 TP ranks. The lowconc key's scenario still uses dram-utilization: 0.90, which gives TOTAL_CPU_DRAM_GB 1932. Fix: make the recorded budget match the pinning. Either give the c16 row its own scenario entry with a dram-utilization of about 0.45, or raise the per-rank bytes to 241500000000. Also pin TOTAL_CPU_DRAM_GB in the override, as every other dram override in this file does, so the binder catches a mismatch.
Why this was flagged
The lowconc key at nvidia-master.yaml (row kv-offloading: dram, conc-list: [16], srt-recipe: agentic.yaml) sits under the scenario's dram-utilization: 0.90. agentic_dram_offload_gb (infx/matrix/generate.py:475-518) turns that into TOTAL_CPU_DRAM_GB=1932 for the 8-GPU point. That is the same value the DEP8 overrides pin, and they allocate exactly 8 x 241500000000 bytes. The new override_tp8_c16 (agentic.yaml:159) allocates 8 x 120000000000 bytes, about 960 GB. infx/results/agentic/__init__.py:205 publishes allocated_cpu_dram_gb from TOTAL_CPU_DRAM_GB, so the c16 point is reported with about twice the DRAM it actually used. The base branch ran c16 with KV_OFFLOADING: none, so it recorded no DRAM budget at all. The binder's guard (single_node.py:145) only compares TOTAL_CPU_DRAM_GB when the recipe sets it. Unlike override_dep8_c32..c192, override_tp8_c16 does not set it, so the guard never fires.
Verification: Trigger: the lowconc key's c16 DRAM row runs. The result's allocated_cpu_dram_gb then reads 1932 GB, but the server pins only about 960 GB. The new override_tp8_c16 in agentic.yaml (line 159) sets "cpu_bytes_to_use_per_rank":120000000000 across tensor-parallel-size 8. Its benchmark.env does not pin TOTAL_CPU_DRAM_GB, so infx/srt_slurm/single_node.py:145 never fires.
| reasoning-parser: deepseek_v4 | ||
| served-model-name: deepseek-ai/DeepSeek-V4-Pro-0813 | ||
| speculative-config: '{"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.77}' | ||
| tensor-parallel-size: 2 |
There was a problem hiding this comment.
🟡 (optional) The new TP2xDP4 recipes hard-code a synthetic acceptance length of 3.77, which AGENTS.md forbids. These are the only recipes in the repo that pin synthetic_acceptance_length. The value is not taken from the measured golden curves. The same speculative-config line appears in agentic-tp2dp4-c48.yaml:54. Fix: in both files, drop rejection_sample_method and synthetic_acceptance_length so speculative-config matches agentic.yaml ({"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"probabilistic"}). The srt driver then injects the golden AL from infx/golden_al_distribution/.
Why this was flagged
AGENTS.md (Launchers and hooks) says: "Never hard-code synthetic acceptance lengths; the srt driver picks measured values from infx/golden_al_distribution/". golden_al_distribution/README.md:152 also says recipes must not add vLLM synthetic_acceptance_length. agentic-tp2dp4-c64.yaml:55 and agentic-tp2dp4-c48.yaml:54 both set "rejection_sample_method":"synthetic","synthetic_acceptance_length":3.77. A repo-wide grep finds no other recipe that sets it. synthetic_acceptance.py:231-241 overwrites the value when a golden AL exists and otherwise switches to block rejection. So the literal only works by accident, and the recipe misstates the behaviour it encodes. The sibling recipe agentic.yaml leaves both fields unset.
Verification: nit. This breaks a rule the repo writes down, but at runtime the driver overwrites or removes the pinned value. AGENTS.md:98 says "Never hard-code synthetic acceptance lengths". agentic-tp2dp4-c64.yaml:54 and agentic-tp2dp4-c48.yaml:54 both set synthetic_acceptance_length to 3.77. No test lints recipes for this field, so CI will not catch it.
| # DEP8 SimpleCPU + DSpark6; c32 runs TP4 x DP2 expert-parallel | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: draft_model, conc-list: [32, 64, 96, 128, 160, 192], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml } | ||
|
|
||
| dsv4-fp4-b200-vllm-agentic-mtp-lowconc: | ||
| image: vllm/vllm-openai@sha256:d428631bb917a33d22308265d1b8be91522f862bf9fd133bee62dde3313664ae | ||
| model: deepseek-ai/DeepSeek-V4-Pro-0813 | ||
| model-prefix: dsv4 | ||
| runner: cluster:b200-nscale | ||
| precision: fp4 | ||
| framework: vllm | ||
| multinode: false | ||
| srt-recipe-dir: dsv4/vllm/b200-fp4-mtp | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.90 | ||
| search-space: | ||
| - { tp: 8, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 4], srt-recipe: agentic-lowconc.yaml } | ||
| # TP8 c10 and c16 use the agentic.yaml overrides; c16 adds SimpleCPU KV offload | ||
| - { tp: 8, kv-offloading: none, spec-decoding: draft_model, conc-list: [10], srt-recipe: agentic.yaml } | ||
| - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: draft_model, conc-list: [16], srt-recipe: agentic.yaml } | ||
| # TP2 x DP4 expert-parallel SimpleCPU + DSpark6 at c64 | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: draft_model, conc-list: [64], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic-tp2dp4-c64.yaml } | ||
| # TP2 x DP4 expert-parallel SimpleCPU + DSpark6 at c48 |
There was a problem hiding this comment.
🟡 nit (optional): The new master-config comments only describe the rows below them, which the AGENTS.md Comments rule forbids. Examples: "TP2 x DP4 expert-parallel SimpleCPU + DSpark6 at c64" and "TP8 c10 and c16 use the agentic.yaml overrides; c16 adds SimpleCPU KV offload". Fix: remove the added comments at nvidia-master.yaml:906 (the "; c32 runs TP4 x DP2" suffix), :923, :926 and :928.
Why this was flagged
AGENTS.md "Comments (mandatory, every file)" bans header comments that describe a config or section. It also bans comments that restate the code. nvidia-master.yaml:923, :926 and :928 are new comments that only restate the tp/ep/conc/srt-recipe fields of the next row. The diff extends :906 with "; c32 runs TP4 x DP2 expert-parallel", which restates the recipe's tensor/data parallel sizes. None of them gives a technical reason for a value.
Verification: The rule is in AGENTS.md lines 54-58, and it says "No header comments describing a file, topology, config, recipe, override, or section". Each comment added in inferencex-e2e/configs/nvidia-master.yaml describes the topology or section of the rows below it. None gives a reason a value was chosen. Nothing breaks at runtime.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
hshrivastava-droid
changed the title
|
/use 40585790288 |
|
@hshrivastava-droid staged run 40585790288. This run remains available across future |
Summary
dsv4-fp4-b200-vllm-agentic-mtp,dsv4-fp4-b200-vllm-agentic-mtp-lowconcvllm/vllm-openai:nightly-8e2beaa3bff595577a069d49d96bde9074294ec2@sha256:3207cb99c0f59b92f7d47a74da8c6eb5467565d811a4f8d9df51df5188f59c70,vllm/vllm-openai@sha256:d428631bb917a33d22308265d1b8be91522f862bf9fd133bee62dde3313664aeAI model disclosure
claude-sonnet-5-5)PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
dsv4-fp4-b200-vllm-agentic-mtp、dsv4-fp4-b200-vllm-agentic-mtp-lowconcvllm/vllm-openai:nightly-8e2beaa3bff595577a069d49d96bde9074294ec2、vllm/vllm-openai@sha256:d428631b…🤖 Generated with Claude Code