[https://nvbugs/5945081][fix] un-waive DeepSeek-V3-Lite NVFP4 pp4 CUTLASS test - #17301
[https://nvbugs/5945081][fix] un-waive DeepSeek-V3-Lite NVFP4 pp4 CUTLASS test#17301xxi-nv wants to merge 1 commit into
Conversation
…LASS test The bug's signature is "Test terminated unexpectedly" -- process death rather than an accuracy assertion. Attempting to reproduce it against TOT on an exclusive 8-GPU B200 node gave three consecutive passes (GSM8K 64.59 / 63.12 / 64.75, ~7 min each), with every iteration verified to have actually exercised the bug path: four distinct ranks reporting ep=1 pp=4 tp=1 world=4, the NVFP4 path taken, GSM8K evaluated to completion, and no stack dumps. Runs that hit an environment problem instead of the bug were excluded rather than counted. Three passes on a flaky process death establish "not currently reproducible", not "fixed", so this un-waive hands the sampling to CI: the case is in the post-merge lists for DGX_B300, GB200, GB300 and RTXPro6000D, and every post-merge run now adds evidence. If it flakes again the waive can be restored with a fresh occurrence to analyze, which is more useful than keeping a possibly-stale waive indefinitely. Note that B200 no longer covers this case -- it was dropped from l0_dgx_b200.yml by NVIDIA#15624, so the stage where the bug was originally reported (DGX_B200-4_GPUs-PyTorch-Post-Merge-1) does not run it anymore. Signed-off-by: xxi <xxi@nvidia.com>
|
/bot run --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1,DGX_B300-4_GPUs-PyTorch-Post-Merge-2,GB200-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GB300-4_GPUs-PyTorch-Post-Merge-3,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2" |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
WalkthroughThe change removes the DeepSeekV3Lite NVFP4 CUTLASS PP4 test from the integration-test waiver list. Changes
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #64002 [ run ] triggered by Bot. Commit: |
|
PR_Github #64002 [ run ] completed with state
|
|
/bot run --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1,DGX_B300-4_GPUs-PyTorch-Post-Merge-2,GB200-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GB300-4_GPUs-PyTorch-Post-Merge-3,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2" |
|
PR_Github #64012 [ run ] triggered by Bot. Commit: |
|
PR_Github #64012 [ run ] completed with state
|
brnguyen2
left a comment
There was a problem hiding this comment.
Verified the coverage claims: the case is in l0_dgx_b300.yml:47, l0_gb200_multi_gpus.yml, l0_gb300_multi_gpus.yml, l0_rtx_pro_6000.yml (all post_merge pytorch blocks) and absent from l0_dgx_b200.yml, so the targeted stages are the right ones. Evidence and scoping are honest. One follow-up: update/close https://nvbugs/5945081 with a pointer to this PR — leaving it open invites the entry being re-added by triage.
Description
Removes the
waives.txtSKIP entry forwhich was waived under https://nvbugs/5945081.
Why
The bug's signature is
Test terminated unexpectedly— process death, not anaccuracy assertion. I attempted to reproduce it against TOT and it did not
reproduce:
Reproduction environment: TOT
0a6d932303, exclusive 8-GPU B200 node, the exactwaived parametrization, run with
pytest -o addopts=''so the waive itself didnot skip the run. Each iteration was verified to have actually exercised the bug
path before being counted — 4 distinct ranks reporting
ep=1 pp=4 tp=1 world=4,the NVFP4 quantization path taken, GSM8K evaluated to completion, and zero stack
dumps. Runs that hit an environment problem (a shared node with a carved NVSwitch
fabric, an NVRTC include-path failure) were classified as invalid and excluded
rather than counted as passes.
Scope note
Three passes on a flaky process-death bug support "not currently reproducible",
not "proven fixed". This PR un-waives so that CI itself becomes the sampler: the
test is in the post-merge lists for DGX_B300, GB200, GB300 and RTXPro6000D, so
each post-merge run adds evidence. If it flakes again, the waive can be restored
with a fresh occurrence to analyze — which is more informative than leaving a
possibly-stale waive in place indefinitely.
Note that B200 no longer runs this case: it was dropped from
l0_dgx_b200.ymlby #15624, so the stage where the bug was originally seen
(
DGX_B200-4_GPUs-PyTorch-Post-Merge-1) does not cover it anymore.Test Coverage
Targeted post-merge stages triggered on this PR (all shards of every list that
contains the case, since shard assignment is decided at runtime):
PR Checklist
pre-commit run --all-filespassesDev Engineer Review
DeepSeekV3LiteNVFP4 CUTLASS PP4 waiver fornvbugs/5945081.waives.txtformat.QA Engineer Review
tests/integration/test_lists/waives.txt.test-db/orqa/files were modified.