[https://nvbugs/6450341][fix] Lower MTP acceptance rate threshold for nvfp4 ws8 trtllm by MrGeva · Pull Request #16375 · NVIDIA/TensorRT-LLM

MrGeva · 2026-07-14T13:02:31Z

Summary

TestNemotronSuperV3::test_mtp[nvfp4_ws8_80gb-trtllm] intermittently fails just under the 50% acceptance rate threshold (observed 49.70%).
The 8-rank all-reduce/all-to-all in this config is not bit-reproducible, so tiny logit noise occasionally flips an argmax and shifts the measured MTP acceptance rate below the current threshold.
Lower the threshold to 0.492 specifically for model_id == "nvfp4", world_size == 8, attn_backend == "trtllm"; all other test_mtp parametrizations keep the existing 0.50 threshold since they run with more margin.

Test plan

Re-run test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[nvfp4_ws8_80gb-trtllm] a few times in CI to confirm it no longer flakes at the new threshold.
Confirm other test_mtp parametrizations (bf16/fp8 ws4, fp8 ws8) are unaffected.

Dev Engineer Review

Lowered the acceptance-rate threshold only for nvfp4_ws8_80gb-trtllm from 0.50 to 0.492.
Preserved the 0.50 threshold for all other test_mtp configurations.
Removed the corresponding waiver.
The change matches the reported non-bit-reproducible 8-rank reduction behavior.

QA Engineer Review

Modified TestNemotronSuperV3::test_mtp.
Removed its entry from tests/integration/test_lists/waives.txt.
No test-db/ or qa/ coverage entry was provided.
Verdict: needs follow-up.

coderabbitai · 2026-07-14T13:04:45Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1b6632a5-eab6-40fa-87ea-0b10eff67fe7

📥 Commits

Reviewing files that changed from the base of the PR and between 1ddd407 and 6e65995.

📒 Files selected for processing (2)

tests/integration/defs/accuracy/test_llm_api_autodeploy.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

tests/integration/defs/accuracy/test_llm_api_autodeploy.py

Walkthrough

The MTP speculative decoding accuracy test now uses a lower acceptance-rate threshold for one NVFP4, eight-rank TRT-LLM configuration. Its test waiver is removed. Other configurations retain the existing threshold.

Changes

MTP accuracy validation

Layer / File(s)	Summary
Configuration-specific acceptance threshold `tests/integration/defs/accuracy/test_llm_api_autodeploy.py`, `tests/integration/test_lists/waives.txt`	The test uses `0.492` for the targeted configuration and `0.50` otherwise. The matching waiver is removed.

Estimated code review effort: 2 (Simple) | ~5 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#17217: Removes a waiver for a different MTP test configuration.
NVIDIA/TensorRT-LLM#17309: Removes waivers for other Nemotron V3 Super integration tests.
NVIDIA/TensorRT-LLM#17340: Adjusts test waivers in the same waiver list.

Suggested reviewers: brnguyen2, zhanruisunch, nvchenghaoz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the NVBugs fix and the specific MTP acceptance-rate threshold change.
Description check	✅ Passed	The description explains the intermittent failure, the targeted threshold change, and the relevant test coverage.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

MrGeva · 2026-07-14T13:06:22Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-07-14T13:11:49Z

PR_Github #59208 [ run ] triggered by Bot. Commit: afe2659 Link to invocation

tensorrt-cicd · 2026-07-14T13:34:01Z

PR_Github #59208 [ run ] completed with state FAILURE. Commit: afe2659
/LLM/main/L0_MergeRequest_PR pipeline #47706 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

jieli-matrix

plz also un-waive the test case when fix landing, thx!

TestNemotronSuperV3::test_mtp[nvfp4_ws8_80gb-trtllm] intermittently fails just under the 50% acceptance rate threshold (observed as low as 49.70%). The 8-rank all-reduce/all-to-all is not bit-reproducible, so tiny logit noise occasionally flips an argmax and shifts the measured acceptance rate. Lower the threshold to 0.492 for this specific config only; other model_id/world_size/attn_backend combinations keep the existing 0.50 threshold since they run with more margin. Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>

Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>

MrGeva · 2026-07-15T08:19:56Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-07-15T08:26:51Z

PR_Github #59405 [ run ] triggered by Bot. Commit: 1c103aa Link to invocation

tensorrt-cicd · 2026-07-15T09:32:12Z

PR_Github #59405 [ run ] completed with state SUCCESS. Commit: 1c103aa
/LLM/main/L0_MergeRequest_PR pipeline #47878 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

MrGeva · 2026-07-15T10:51:11Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

…akiness Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>

MrGeva · 2026-07-16T04:39:01Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-07-16T04:45:35Z

PR_Github #59607 [ run ] triggered by Bot. Commit: 5c2f420 Link to invocation

tensorrt-cicd · 2026-07-16T14:36:39Z

PR_Github #59607 [ run ] completed with state SUCCESS. Commit: 5c2f420
/LLM/main/L0_MergeRequest_PR pipeline #48048 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…akiness Signed-off-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com>

galagam · 2026-07-16T21:17:00Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-07-16T21:23:21Z

PR_Github #59785 [ run ] triggered by Bot. Commit: 5a60145 Link to invocation

tensorrt-cicd · 2026-07-16T21:38:04Z

PR_Github #59785 [ run ] completed with state FAILURE. Commit: 5a60145

Link to invocation

…akiness Signed-off-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com>

coderabbitai · 2026-08-06T05:35:06Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

galagam · 2026-08-06T05:35:34Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-08-06T05:41:23Z

PR_Github #64237 [ run ] triggered by Bot. Commit: 6e65995 Link to invocation

tensorrt-cicd · 2026-08-06T08:18:30Z

PR_Github #64237 [ run ] completed with state FAILURE. Commit: 6e65995
/LLM/main/L0_MergeRequest_PR pipeline #52143 completed with status: 'FAILURE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

galagam · 2026-08-06T09:32:45Z

/bot run

tensorrt-cicd · 2026-08-06T09:38:14Z

PR_Github #64297 [ run ] triggered by Bot. Commit: 6e65995 Link to invocation

tensorrt-cicd · 2026-08-06T10:07:45Z

PR_Github #64297 [ run ] completed with state FAILURE. Commit: 6e65995
/LLM/main/L0_MergeRequest_PR pipeline #52197 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

MrGeva requested review from a team as code owners July 14, 2026 13:02

MrGeva requested review from LarryXFly, nvchenghaoz and xinhe-nv July 14, 2026 13:02

github-actions Bot assigned MrGeva Jul 14, 2026

MrGeva changed the title ~~[None][fix] Lower MTP acceptance rate threshold for nvfp4 ws8 trtllm~~ [https://nvbugs.com/6450341][fix] Lower MTP acceptance rate threshold for nvfp4 ws8 trtllm Jul 14, 2026

MrGeva changed the title ~~[https://nvbugs.com/6450341][fix] Lower MTP acceptance rate threshold for nvfp4 ws8 trtllm~~ [https://nvbugs/6450341][fix] Lower MTP acceptance rate threshold for nvfp4 ws8 trtllm Jul 14, 2026

govind-ramnarayan approved these changes Jul 14, 2026

View reviewed changes

jieli-matrix requested changes Jul 15, 2026

View reviewed changes

LarryXFly approved these changes Jul 15, 2026

View reviewed changes

MrGeva force-pushed the fix-nemotron-super-v3-mtp-acceptance-rate-flakiness branch from afe2659 to b6ce009 Compare July 15, 2026 07:57

unwaive

1c103aa

Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>

MrGeva requested review from a team as code owners July 15, 2026 08:18

MrGeva requested review from BowenFu, ZhanruiSunCh, brnguyen2, jieli-matrix, tburt-nv and yingguo-trt July 15, 2026 08:18

BowenFu approved these changes Jul 15, 2026

View reviewed changes

Merge branch 'main' into fix-nemotron-super-v3-mtp-acceptance-rate-fl…

5c2f420

…akiness Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>

MrGeva enabled auto-merge (squash) July 15, 2026 11:55

tburt-nv approved these changes Jul 15, 2026

View reviewed changes

jieli-matrix approved these changes Jul 16, 2026

View reviewed changes

yingguo-trt approved these changes Jul 16, 2026

View reviewed changes

Merge branch 'main' into fix-nemotron-super-v3-mtp-acceptance-rate-fl…

5a60145

…akiness Signed-off-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com>

xinhe-nv approved these changes Jul 20, 2026

View reviewed changes

Merge branch 'main' into fix-nemotron-super-v3-mtp-acceptance-rate-fl…

6e65995

…akiness Signed-off-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com>

Uh oh!

Conversation

MrGeva commented Jul 14, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Dev Engineer Review

QA Engineer Review

Uh oh!

coderabbitai Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

MrGeva commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

tensorrt-cicd commented Jul 14, 2026

Uh oh!

jieli-matrix left a comment

Choose a reason for hiding this comment

Uh oh!

MrGeva commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

MrGeva commented Jul 15, 2026

Uh oh!

MrGeva commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

galagam commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

galagam commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

galagam commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

MrGeva commented Jul 14, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading