[https://nvbugs/6564714][fix] Fix the FP8BlockScaleMoERunner get_fallback_tactic by leslie-fang25 · Pull Request #17328 · NVIDIA/TensorRT-LLM

leslie-fang25 · 2026-08-06T01:11:38Z

Dev Engineer Review

Updated FP8 MoE tactic cache keys and fallback validation to include num_fused_shared_experts.
This prevents tactic reuse across different fused shared-expert configurations.
The change preserves the existing API and limits scope to tactic selection.
Removed waiver entries associated with nvbugs/6564714.
Added two GLM4 MoE equivalence waivers for nvbugs/6566730.
The test-list format and bug references are consistent.

QA Engineer Review

No test code changed.
Removed the DeepSeek V3 Lite host-performance waiver and 19 single-GPU AutoDeploy MoE waiver entries.
Added two GLM4 MoE equivalence waiver entries.
No test-db/ or qa/ files were modified.
CBTS coverage data is unavailable.

Verdict: needs follow-up.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

leslie-fang25 · 2026-08-06T01:12:28Z

/bot run --disable-fail-fast

coderabbitai · 2026-08-06T01:14:13Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6c1b7b8d-2792-4756-8a84-538802efc929

📥 Commits

Reviewing files that changed from the base of the PR and between 409bccf and 622eda1.

📒 Files selected for processing (2)

tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py

Walkthrough

The FP8 MoE autotuner and fallback-tactic cache now include num_fused_shared_experts. GLM4 MoE decoder-layer and equivalence tests now have integration-test waiver entries.

Changes

FP8 MoE tactic selection

Layer / File(s)	Summary
Fused shared-expert tactic validation `tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py`, `tests/integration/test_lists/waives.txt`	Cache identities and fallback validation now include `num_fused_shared_experts`. Two GLM4 MoE test skip entries were added.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#15297: Extends FP8 fused shared-expert support in the same custom-operation path.
NVIDIA/TensorRT-LLM#17120: Modifies FP8 MoE fallback-tactic handling in the same file.
NVIDIA/TensorRT-LLM#17335: Modifies related integration-test waiver entries.

Suggested reviewers: qijune, schetlur-nv, emmaqiaoch

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description includes the template but provides no issue summary, solution details, or test coverage information.	Add a concise Description section explaining the issue and solution, and list the relevant tests and their coverage.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title identifies the NVBug, fix type, and affected FP8BlockScaleMoERunner fallback tactic, matching the main code change.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py`:
- Around line 925-926: Update FP8BlockScaleMoERunner.unique_id() to include
num_fused_shared_experts, matching the fallback cache key and
get_valid_configs() inputs. Add a regression test that creates configurations
with different fused shared-expert counts and verifies their unique
IDs/autotuner cache entries remain distinct.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ae12afc7-2894-44fa-a694-55418aed9cbc

📥 Commits

Reviewing files that changed from the base of the PR and between 1310036 and 610cf60.

📒 Files selected for processing (2)

tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

leslie-fang25 · 2026-08-06T01:30:42Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-06T01:34:03Z

PR_Github #64152 [ run ] triggered by Bot. Commit: 64a2b60 Link to invocation

tensorrt-cicd · 2026-08-06T01:36:54Z

PR_Github #64154 [ run ] triggered by Bot. Commit: 64a2b60 Link to invocation

tensorrt-cicd · 2026-08-06T01:43:09Z

PR_Github/17328-64a2b60 #64152 was force-killed by a newer pipeline run.
L0 job information not available (job may not have been triggered yet).

Link to superseding invocation

brnguyen2

Approving — the comments below are optional touch-ups, not blockers.

Fix looks right: the key order now matches FP8BlockScaleMoeRunner::getValidConfigs in cpp/tensorrt_llm/thop/fp8BlockScaleMoe.cpp:395 and get_valid_tactics at [tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py:967](https://github.com/NVIDIA/TensorRT-LLM/pull/17328/files#diff-fe56cea5ab264b0896bbabd0288bbc3f6ffc18469c8c7fe771142be2bce8d022R967). Including it in unique_id() is also needed on its own, since the C++ side restricts tileN when fused shared experts are present — tactic validity genuinely depends on it.

Please confirm the two re-enabled AutoDeploy tests actually pass in this PR's B200 stage before merging, rather than only inferring it from the mechanism.

coderabbitai · 2026-08-06T05:32:48Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

leslie-fang25 · 2026-08-06T05:33:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-06T05:39:13Z

PR_Github #64235 [ run ] triggered by Bot. Commit: d8247c7 Link to invocation

tensorrt-cicd · 2026-08-06T05:42:10Z

PR_Github #64154 [ run ] completed with state ABORTED. Commit: 64a2b60

Link to invocation

…back_tactic Signed-off-by: leslief <leslief@nvidia.com>

Signed-off-by: leslief <leslief@nvidia.com>

coderabbitai · 2026-08-06T05:57:13Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

leslie-fang25 · 2026-08-06T05:57:21Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-06T06:02:53Z

PR_Github #64241 [ run ] triggered by Bot. Commit: 622eda1 Link to invocation

tensorrt-cicd · 2026-08-06T06:08:26Z

PR_Github #64235 [ run ] completed with state ABORTED. Commit: d8247c7

Link to invocation

tensorrt-cicd · 2026-08-06T10:00:58Z

PR_Github #64241 [ run ] completed with state FAILURE. Commit: 622eda1
/LLM/main/L0_MergeRequest_PR pipeline #52147 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

leslie-fang25 · 2026-08-06T10:01:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-06T10:08:05Z

PR_Github #64300 [ run ] triggered by Bot. Commit: 622eda1 Link to invocation

tensorrt-cicd · 2026-08-06T12:22:23Z

PR_Github #64300 [ run ] completed with state SUCCESS. Commit: 622eda1
/LLM/main/L0_MergeRequest_PR pipeline #52199 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

leslie-fang25 · 2026-08-06T12:35:21Z

test_e2e[aggr_upload-host_perf_deepseek_v3_lite-v3lite_fp8_bs8_128_256]
unittest/auto_deploy/singlegpu/custom_ops/moe/test_trtllm_moe.py
unittest/auto_deploy/singlegpu/transformations/library/test_moe_fusion.py
passes in https://prod.blsm.nvidia.com/sw-tensorrt-llm-github-3/blue/organizations/jenkins/LLM%2Fmain%2FL0_Test-x86_64-Single-GPU/detail/L0_Test-x86_64-Single-GPU/5431/tests

leslie-fang25 requested review from a team as code owners August 6, 2026 01:11

leslie-fang25 requested review from EmmaQiaoCh, QiJune, cascade812, schetlur-nv, tburt-nv, tongyuantongyu and yingguo-trt August 6, 2026 01:11

github-actions Bot assigned leslie-fang25 Aug 6, 2026

coderabbitai Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py

yingguo-trt approved these changes Aug 6, 2026

View reviewed changes

brnguyen2 approved these changes Aug 6, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/custom_ops/trtllm_gen_custom_ops.py

Wanli-Jiang approved these changes Aug 6, 2026

View reviewed changes

tongyuantongyu approved these changes Aug 6, 2026

View reviewed changes

tongyuantongyu mentioned this pull request Aug 6, 2026

[None][infra] Waive 18 failed cases for main in post-merge 2887 #17335

Merged

leslie-fang25 force-pushed the leslie/fix_6564714 branch from 64a2b60 to d8247c7 Compare August 6, 2026 05:32

coderabbitai Bot mentioned this pull request Aug 6, 2026

[https://nvbugs/6525059][fix] Drop 8 from FP8BlockScaleMoeRunner::mSupportedTileN, covering both the… #17135

Open

2 tasks

leslie-fang25 added 2 commits August 6, 2026 05:56

[https://nvbugs/6564714][fix] Fix the FP8BlockScaleMoERunner get_fall…

34b9784

…back_tactic Signed-off-by: leslief <leslief@nvidia.com>

address comments

622eda1

Signed-off-by: leslief <leslief@nvidia.com>

leslie-fang25 force-pushed the leslie/fix_6564714 branch from d8247c7 to 622eda1 Compare August 6, 2026 05:57

coderabbitai Bot mentioned this pull request Aug 6, 2026

[None][fix] Pass num_fused_shared_experts in FP8 MoE fallback-tactic path #17362

Open

1 task

leslie-fang25 merged commit bc334fb into NVIDIA:main Aug 6, 2026
10 checks passed

Uh oh!

Conversation

leslie-fang25 commented Aug 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

leslie-fang25 commented Aug 6, 2026

Uh oh!

coderabbitai Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leslie-fang25 commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

brnguyen2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

leslie-fang25 commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

leslie-fang25 commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

leslie-fang25 commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

leslie-fang25 commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

leslie-fang25 commented Aug 6, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 6, 2026 •

edited

Loading

leslie-fang25 commented Aug 6, 2026 •

edited

Loading