From 78b8e6452e4db9a37c0dd4d37b27a78bd6331fce Mon Sep 17 00:00:00 2001 From: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> Date: Thu, 6 Aug 2026 07:07:17 -0700 Subject: [PATCH] [https://nvbugs/6523520][fix] Halve gb200 r1-fp4 128k8k con128 multi_round to fit perf-sanity budget The aggr_upload-ctx_only-gb200_deepseek-r1-fp4_128k8k_con128 case (and its e2e / gen_only co-tenants sharing the same config) times out under the 180-min perf-sanity budget. Root cause is the second measurement round, not a functional slowdown. ctx_only forces OSL=1 and iterations=multi_round, so the run is concurrency x multi_round = 128 x 2 = 256 sequential 131072-token (128k) prefills through a max_batch_size=1, pipeline_parallel_size=8 context server. Reducing to a single round (128 prefills) makes ctx_only finish in ~18.5 min -- and 4x the prefill count (32 -> 128) added only ~10% wall-clock, confirming the run time is dominated by fixed server startup/warmup, not prefill count. So the original 256-prefill config was not merely large: its second round livelocked and consumed the full 180-min wall-clock cap. Fix (config-only, shared NIXL yaml): multi_round 2 -> 1. Concurrency is left at 128 (the representative workload); num-prompts drops 256 -> 128. Verified on lyris GB200 (rc24 wheel), both cases sharing this config pass far under the 180-min TIMEOUT: - ctx_only : PASSED, 1 passed in 1112.18s / 0:18:32 (job elapsed 23m) - e2e : PASSED, 1 passed in 2534.09s / 0:42:14 (job elapsed 49m) Un-waive both cases that were skipped for the now-resolved timeout (the anchors 6418510 / 6426890 are closed post-merge batch bugs): - ctx_only (was SKIP nvbugs/6418510) - e2e (was SKIP nvbugs/6426890) Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> --- tests/integration/test_lists/waives.txt | 2 -- ...4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index b2971766eacd..c09f6ed64ddd 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -354,7 +354,6 @@ llmapi/test_llm_api_pytorch_bart.py::test_bart_pytorch_generate_encoder_decoder_ llmapi/test_llm_api_pytorch_moe_lora.py::test_mixtral_moe_routed_expert_fp8_multi_lora_varying_ranks[cudagraph] SKIP (https://nvbugs/6463829) llmapi/test_llm_api_pytorch_moe_lora.py::test_mixtral_moe_routed_expert_fp8_multi_lora_varying_ranks[eager] SKIP (https://nvbugs/6463829) llmapi/test_llm_examples.py::test_llmapi_speculative_decoding_eagle3 SKIP (https://nvbugs/6075431) -perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL] SKIP (https://nvbugs/6418510) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL] SKIP (https://nvbugs/6517846) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6517846) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6517846) @@ -369,7 +368,6 @@ perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_ perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_mtp3_8k1k] SKIP (https://nvbugs/6535767) perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_tp4_8k1k] SKIP (https://nvbugs/6535767) perf/test_perf_sanity.py::test_e2e[aggr_upload-super_ad_blackwell-super_ad_ws1_1k1k] SKIP (https://nvbugs/6153575) -perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL] SKIP (https://nvbugs/6426890) perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL] SKIP (https://nvbugs/6550133) perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL] SKIP (https://nvbugs/6550133) perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL] SKIP (https://nvbugs/6566777) diff --git a/tests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml b/tests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml index e7c09700e588..e493891d0af4 100644 --- a/tests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml +++ b/tests/scripts/perf-sanity/disaggregated/gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml @@ -17,7 +17,7 @@ slurm: benchmark: mode: e2e use_nv_sa_benchmark: false - multi_round: 2 + multi_round: 1 benchmark_ratio: 0.0 streaming: true concurrency_list: '128'