[None][infra] Unwaive perf-sanity disagg gen_only cases

[chenfeiz0326]

Summary

Removes all 20 perf/test_perf_sanity.py disagg gen_only SKIP waivers so post-merge re-runs those cases against the now-fixed harness.

Why these were waived

Builds 2873 / 2884 / 2876 recorded results-timeout.xml "Test terminated unexpectedly" for these cases (nvbugs 6541361 / 6542166 / 6561566 / 6550133, and the duplicate 6551021). These were not workload failures — the gen_only benchmark passed end-to-end (all ranks exit 0, "N passed", benchmark_status="Done", 0 failed requests), then the GEN slurm step idled ~76–78 min in teardown until the 90-min stage watchdog killed it.

Root cause (already fixed on main)

wait_for_gen_log_sentinels(), added in 99bdffc4c3 (#16717), polled gen_server_{i}.done bounded by DEFAULT_TIMEOUT (10800s) — longer than the 5400s stage watchdog — so a stuck/un-reaped multi-node gen srun turned an optional log-flush sync into a stage cancellation. A companion defect derived testOutputDir from a positional test-list index that could disagree with the pytest-split shard, so the sentinel was polled in the wrong directory.

Both are merged to main:

• [None][fix] Align perf launcher with pytest shard #17134 (721c46b8c2) — align the launcher's testOutputDir with the pytest shard.
• [None][fix] Bound GEN log sentinel wait #17140 (22184ba7cb) — bound the wait to GEN_LOG_SENTINEL_TIMEOUT (120s, well below the watchdog), fall back to parsing the current gen log on timeout, and gate the whole path on benchmark_mode == "gen_only".

Scope / caveat

• Removes only disagg gen_only perf-sanity waivers (20 lines). Non-gen_only perf-sanity waivers (ctx_only, e2e, aggregated) are left untouched.
• A few removed lines cite bugs that also tracked genuine gen_only failures (6490049 gen_only Server/Client error; 6374872). If those recur, the post-merge run will re-file an accurate, harness-fix-independent waiver — which is the correct signal, since the current waiver conflates them with the teardown hang.

Testing

Per the NVBugs waiver-removal guidance, the affected post-merge stages must be re-run rather than skipped, e.g.:

/bot run --stage-list "GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3 GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

(full stage list follows from the 20 unwaived cases).

Dev Engineer Review

• Removed 20 gen_only skip waivers from tests/integration/test_lists/waives.txt.
• The entries cover disaggregated-upload configurations for B200 and GB300 DeepSeek, GPT-OSS, Kimi, Qwen, and GLM tests.
• The change has no production code or public API impact.
• The waiver-file scope and format are consistent with the stated objective.
• The waivers can be removed because the related harness issues were fixed in #17134 and #17140.

QA Engineer Review

• Modified tests/integration/test_lists/waives.txt.
• Removed 20 perf/test_perf_sanity.py disaggregated-upload gen_only entries.
• No test-db/ or qa/ files were modified.
• Verdict: needs follow-up. CBTS coverage data is unavailable.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a8a02446-0d6a-4d7f-8e70-e79a903630fd

📥 Commits

Reviewing files that changed from the base of the PR and between a1c13ff and 9b4617a.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

---

Walkthrough

Removes 20 gen_only skip entries for disaggregated-upload performance sanity tests covering B200 and GB300 model configurations.

Changes

Cohort / File(s)	Summary
Performance test waiver cleanup tests/integration/test_lists/waives.txt	Removes 20 disaggregated-upload waiver entries for DeepSeek, GPT-OSS, Kimi, Qwen, and GLM configurations.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#17217: Removes other test waivers from the same waiver list.
• NVIDIA/TensorRT-LLM#17272: Adjusts disaggregated performance test waivers in the same file.
• NVIDIA/TensorRT-LLM#17338: Updates related disaggregated performance sanity-test waivers.

Suggested reviewers: brnguyen2, bowenfu

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required ticket and type format and clearly describes removal of performance sanity waivers.
Description check	✅ Passed	The description explains the change, root cause, scope, fixes, caveats, and required post-merge testing in sufficient detail.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #64279 Bot args parsing error: usage: /bot [-h]
{run,kill,skip,submit,reviewers,reuse-pipeline,reuse-review} ...
/bot: error: unrecognized arguments: These 20 stages are the post-merge disagg gen_only perf-sanity cases unwaived by this PR (one case per stage; -k = pytest-split least_duration group index against tests/integration/defs/.test_durations). Running them here re-validates the teardown-hang fix (#17134 + #17140) against ToT before the waivers are dropped.

Link to invocation

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-6,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-7,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-8,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

[tensorrt-cicd]

PR_Github #64283 [ run ] triggered by Bot. Commit: 9b4617a Link to invocation

[BowenFu]

16 of the 20 removals are well-evidenced and I'd take them as-is — #17134 (721c46b8c2) and #17140 (22184ba7cb) are both on main, the sentinel wait really is bounded at GEN_LOG_SENTINEL_TIMEOUT=120 s well under the 5400 s watchdog and gated on benchmark_mode == "gen_only" (test_perf_sanity.py:112,1337-1376,1527-1530), and the shard/testOutputDir mismatch is fixed in jenkins/scripts/perf/submit.py:235-275,705-744. The scope is honest too: only gen_only disagg lines go, ctx_only/e2e/aggregated stay.

The four that cite 6374872 and 6490049 are a different case, and I don't think the "post-merge will re-file an accurate waiver" argument covers them. #16717 — the commit that introduced the sentinel and that this PR cites as the root cause — says in its own message that those were deliberately kept:

that still regress independent of the parse race (6478615, 6374872, and three under the 6490049 umbrella) also stay waived.

So the author of the harness change had already separated the teardown hang from these four and concluded they fail for their own reasons. Nothing since 99bdffc4c3 touches those four configurations, so 721c46b8c2/22184ba7cb don't establish that they're fixed — removing them re-reds four cases that are already known-red for a cause this PR doesn't address.

Mitigating, and why I'm not making a big deal of it: all four are stage: post_merge (l0_gb200_multi_nodes_perf_sanity_*, l0_gb300_multi_nodes_perf_sanity_*), so nobody's PR gets blocked either way. Suggestion below restores just those four and leaves the other 16 removed.

[BowenFu]

Suggested change

	perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL] SKIP (https://nvbugs/6490049)
	perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL] SKIP (https://nvbugs/6490049)
	perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL] SKIP (https://nvbugs/6374872)
	perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL] SKIP (https://nvbugs/6490049)
	perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL] SKIP (https://nvbugs/6490049)
	perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL] SKIP (https://nvbugs/6490049)

Restores only the four whose bugs 99bdffc4c3 explicitly kept waived as independent of the sentinel race — 6374872 plus the three under 6490049. The other 16 (6561566, 6550133, and the one 6566777) are genuinely retired by #17134 + #17140 and stay removed.

[tensorrt-cicd]

PR_Github #64283 [ run ] completed with state FAILURE. Commit: 9b4617a
/LLM/main/L0_MergeRequest_PR pipeline #52182 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[brnguyen2]

Approving — the comments below are optional touch-ups, not blockers.

Confirmed both fixes (721c46b8c, 22184ba7c) are ancestors of this head, and #17140's gating on benchmark_mode == "gen_only" matches the scoping here. Two things before merge:

• All four cited bugs are still open. Has the /bot run --stage-list re-run actually been executed and passed? Without that, the merged fix is the only evidence and these come back red post-merge.
• The 6374872 line is the weak one — that bug tracks a capacity-class failure, not the teardown timeout, and no part of #17134/#17140 addresses it. Consider keeping that single waiver until the stage re-run proves otherwise.