Skip to content

[https://nvbugs/5945081][fix] un-waive DeepSeek-V3-Lite NVFP4 pp4 CUTLASS test - #17301

Open
xxi-nv wants to merge 1 commit into
NVIDIA:mainfrom
xxi-nv:fix/unwaive-5945081
Open

[https://nvbugs/5945081][fix] un-waive DeepSeek-V3-Lite NVFP4 pp4 CUTLASS test#17301
xxi-nv wants to merge 1 commit into
NVIDIA:mainfrom
xxi-nv:fix/unwaive-5945081

Conversation

@xxi-nv

@xxi-nv xxi-nv commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Description

Removes the waives.txt SKIP entry for

accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-pp4-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-low_precision_combine=False-torch_compile=False]

which was waived under https://nvbugs/5945081.

Why

The bug's signature is Test terminated unexpectedly — process death, not an
accuracy assertion. I attempted to reproduce it against TOT and it did not
reproduce:

iteration result GSM8K accuracy wall time
1 PASS 64.59 6:46
2 PASS 63.12 6:37
3 PASS 64.75 6:48

Reproduction environment: TOT 0a6d932303, exclusive 8-GPU B200 node, the exact
waived parametrization, run with pytest -o addopts='' so the waive itself did
not skip the run. Each iteration was verified to have actually exercised the bug
path before being counted — 4 distinct ranks reporting ep=1 pp=4 tp=1 world=4,
the NVFP4 quantization path taken, GSM8K evaluated to completion, and zero stack
dumps. Runs that hit an environment problem (a shared node with a carved NVSwitch
fabric, an NVRTC include-path failure) were classified as invalid and excluded
rather than counted as passes.

Scope note

Three passes on a flaky process-death bug support "not currently reproducible",
not "proven fixed". This PR un-waives so that CI itself becomes the sampler: the
test is in the post-merge lists for DGX_B300, GB200, GB300 and RTXPro6000D, so
each post-merge run adds evidence. If it flakes again, the waive can be restored
with a fresh occurrence to analyze — which is more informative than leaving a
possibly-stale waive in place indefinitely.

Note that B200 no longer runs this case: it was dropped from l0_dgx_b200.yml
by #15624, so the stage where the bug was originally seen
(DGX_B200-4_GPUs-PyTorch-Post-Merge-1) does not cover it anymore.

Test Coverage

Targeted post-merge stages triggered on this PR (all shards of every list that
contains the case, since shard assignment is decided at runtime):

DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2,
GB200-4_GPUs-PyTorch-Post-Merge-1,
GB300-4_GPUs-PyTorch-Post-Merge-1, GB300-4_GPUs-PyTorch-Post-Merge-2, GB300-4_GPUs-PyTorch-Post-Merge-3,
RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1, RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2

PR Checklist

  • PR title uses the required format
  • Only one concern per PR
  • pre-commit run --all-files passes

Dev Engineer Review

  • Removed the DeepSeekV3Lite NVFP4 CUTLASS PP4 waiver for nvbugs/5945081.
  • The change uses the expected waives.txt format.
  • The change has no code, API, performance, or error-handling impact.
  • The scope is limited to the intended test waiver.

QA Engineer Review

  • Modified tests/integration/test_lists/waives.txt.
  • Removed one waiver entry.
  • No test-db/ or qa/ files were modified.
  • The test is un-waived for post-merge CI coverage on DGX_B300, GB200, GB300, and RTXPro6000D.
  • Verdict: needs follow-up. CBTS coverage data is unavailable.

…LASS test

The bug's signature is "Test terminated unexpectedly" -- process death rather
than an accuracy assertion. Attempting to reproduce it against TOT on an
exclusive 8-GPU B200 node gave three consecutive passes (GSM8K 64.59 / 63.12 /
64.75, ~7 min each), with every iteration verified to have actually exercised
the bug path: four distinct ranks reporting ep=1 pp=4 tp=1 world=4, the NVFP4
path taken, GSM8K evaluated to completion, and no stack dumps. Runs that hit an
environment problem instead of the bug were excluded rather than counted.

Three passes on a flaky process death establish "not currently reproducible",
not "fixed", so this un-waive hands the sampling to CI: the case is in the
post-merge lists for DGX_B300, GB200, GB300 and RTXPro6000D, and every
post-merge run now adds evidence. If it flakes again the waive can be restored
with a fresh occurrence to analyze, which is more useful than keeping a
possibly-stale waive indefinitely.

Note that B200 no longer covers this case -- it was dropped from l0_dgx_b200.yml
by NVIDIA#15624, so the stage where the bug was originally reported
(DGX_B200-4_GPUs-PyTorch-Post-Merge-1) does not run it anymore.

Signed-off-by: xxi <xxi@nvidia.com>
@xxi-nv

xxi-nv commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1,DGX_B300-4_GPUs-PyTorch-Post-Merge-2,GB200-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GB300-4_GPUs-PyTorch-Post-Merge-3,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2"

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b43cd7f5-53c3-4223-809c-0b1620f179b2

📥 Commits

Reviewing files that changed from the base of the PR and between 9564b3b and 24084cf.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Walkthrough

The change removes the DeepSeekV3Lite NVFP4 CUTLASS PP4 test from the integration-test waiver list.

Changes

Cohort / File(s) Summary
Integration test waiver
tests/integration/test_lists/waives.txt
Removes the skip entry for the DeepSeekV3Lite NVFP4 CUTLASS PP4 test.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: brnguyen2

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title uses the required NVBugs and fix format and clearly states that the DeepSeek-V3-Lite test is un-waived.
Description check ✅ Passed The description explains the change, rationale, reproduction details, scope, CI coverage, and checklist status.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64002 [ run ] triggered by Bot. Commit: 24084cf Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64002 [ run ] completed with state FAILURE. Commit: 24084cf
/LLM/main/L0_MergeRequest_PR pipeline #51936 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@xxi-nv

xxi-nv commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1,DGX_B300-4_GPUs-PyTorch-Post-Merge-2,GB200-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GB300-4_GPUs-PyTorch-Post-Merge-3,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1,RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64012 [ run ] triggered by Bot. Commit: 24084cf Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64012 [ run ] completed with state SUCCESS. Commit: 24084cf
/LLM/main/L0_MergeRequest_PR pipeline #51944 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@brnguyen2 brnguyen2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the coverage claims: the case is in l0_dgx_b300.yml:47, l0_gb200_multi_gpus.yml, l0_gb300_multi_gpus.yml, l0_rtx_pro_6000.yml (all post_merge pytorch blocks) and absent from l0_dgx_b200.yml, so the targeted stages are the right ones. Evidence and scoping are honest. One follow-up: update/close https://nvbugs/5945081 with a pointer to this PR — leaving it open invites the entry being re-added by triage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants