[None][test] Fix perf-core timeouts and 96G-part coverage for large-checkpoint cases by ruodil · Pull Request #17183 · NVIDIA/TensorRT-LLM

ruodil · 2026-08-03T06:18:15Z

Dev Engineer Review

Updated tests/integration/test_lists/qa/llm_perf_core.yml.
Increased timeouts for five llm_perf_core cases.
Added TIMEOUT(120) to the large Qwen 397B FP8 checkpoint-load test and the Qwen 235B FP4 RTX 6000 Server workload.
Moved four DeepSeek R1 FP8 cases from the >90,000 MiB condition to a new >120,000 MiB condition.
Excluded 96G GPUs from cases that require approximately 645G checkpoints.
Retained coverage on H20, B200, and B300 systems with more than 120G per GPU.
No public entities, APIs, or production code changed.
The configuration scope and conditions match the large-checkpoint requirements.

QA Engineer Review

Modified QA file: tests/integration/test_lists/qa/llm_perf_core.yml.
Updated five existing performance-test entries with longer timeouts.
Moved four DeepSeek R1 FP8 entries to the new >120,000 MiB GPU-memory condition.
No test functions changed.
CBTS coverage data is unavailable.
Verdict: needs follow-up.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…ower platforms Five llm_perf_core cases fail with pytest-timeout rather than a real error on the weekly QA run. In each case the harness reports only "Failed: Timeout (>3600.0s) from pytest-timeout" with no exception, and the platforms hitting it are the slowest / smallest-memory members of their condition (RTX 6000 Server, H20-3e), so the wall time is dominated by checkpoint load or by a long low-concurrency decode rather than by a defect. Give those five cases an explicit TIMEOUT so the run reflects real perf signal instead of a truncated one: * llama_v3.3_70b_instruct_fp8 20000,2000 reqs:1000 120 -> 180 min * llama_v3.1_nemotron_ultra_253b bf16 128,128 60 -> 180 min * qwen3.5_397b_a17b_fp8 128,128 ep:8 60 -> 120 min * llama_v3.3_70b_instruct_fp4 8000,1000 reqs:3000 60 -> 120 min * qwen3_235b_a22b_fp4 1000,2000 con:8 60 -> 120 min No cases are added or removed and no other case is affected. Note this only removes the timeout noise; for the two largest checkpoints (474G nemotron_ultra_253b, 380G qwen3.5_397b_a17b_fp8) the load time itself is long because of host-memory checkpoint prefetch, tracked separately in https://nvbugs/6272421 and https://nvbugs/6272397. Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com>

deepseek_r1_0528_fp8 and deepseek_v3.2_fp8 are ~645G checkpoints run at ep:8 tp:8, i.e. ~81G/GPU of weights. On a 96G part (NVML reports ~95.0 GiB usable) that leaves only ~14G per GPU for KV cache, activations and comm buffers, so these four cases cannot produce a meaningful perf number there and fail every weekly run. Condition 9 already gates on gpu_memory (gt: 90000 MiB, which is how 80G H100 is excluded), but a 96G part clears that bar. Split the four cases into a sibling condition with gt: 120000 MiB, which keeps H20-3e (~143.8K MiB), B200 (~183.4K MiB) and B300 (~294.9K MiB) and excludes the 96G part (~97.9K MiB). Effective per-platform case counts, before -> after: H100-SC 99 -> 99 H20-3e 117 -> 117 B200 177 -> 177 B300 177 -> 177 RTX 6000 Server 96 -> 92 (the four cases above) L40S 22 -> 22 Only the 96G part loses coverage; no case is deleted outright, so the larger-memory platforms keep tracking https://nvbugs/6272397. Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com>

coderabbitai · 2026-08-03T06:24:21Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 01211ce1-3814-44b5-828b-af9e6cbaa94e

📥 Commits

Reviewing files that changed from the base of the PR and between 77bc7f0 and 874f3da.

📒 Files selected for processing (1)

tests/integration/test_lists/qa/llm_perf_core.yml

Walkthrough

The LLM performance test list adds condition 10b for high-memory H20, B200, and B300 systems. It moves the large DeepSeek R1 FP8 tests from condition 10 and adds 120-minute timeouts to two large Qwen workloads.

Changes

LLM performance test configuration

Layer / File(s)	Summary
Condition 10b test routing `tests/integration/test_lists/qa/llm_perf_core.yml`	Adds condition 10b for systems with more than 120G per GPU and moves the large DeepSeek R1 FP8 tests from condition 10.
Long-running workload timeouts `tests/integration/test_lists/qa/llm_perf_core.yml`	Adds explanatory comments and 120-minute timeouts to the Qwen 397B FP8 checkpoint-load test and the Qwen 235B FP4 RTX 6000 Server workload.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#17303: Both PRs modify LLM performance conditions and DeepSeek-related test coverage.

Suggested reviewers: larryxfly, yufeiwu-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description retains the template but provides no issue, solution, or test coverage details.	Add a concise Description explaining the issue and solution, and list the relevant test coverage and validation performed.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the timeout fixes and 96G coverage changes in the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/qa/llm_perf_core.yml`:
- Line 15: Add the standard NVIDIA copyright header to the top of the
llm_perf_core.yml file, using the 2026 copyright year, while preserving the
existing QA list content and formatting.
- Around line 377-388: Restrict condition 9b to the intended B200 and B300 GPU
platforms rather than relying only on the per-GPU gpu_memory threshold. Update
the condition under the visible system_gpu_count and compute_capability ranges
by adding the appropriate wildcards.gpu entries or narrowing the predicate,
while preserving the existing memory and capability constraints.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0c74905f-6a5f-4635-ba2a-c0c64a17f899

📥 Commits

Reviewing files that changed from the base of the PR and between 74f3ccf and 272e405.

📒 Files selected for processing (1)

tests/integration/test_lists/qa/llm_perf_core.yml

yufeiwu-nv

llama_v3.3_70b, llama_v3.1_nemotron_ultra_253b are deleted because of PM request. Please rebase the main

Resolve conflict in tests/integration/test_lists/qa/llm_perf_core.yml. main renumbered the condition index (new '8: B300' block pushed the old 8/9/10 to 9/10/11) and deleted several test families outright. Reapplied only the parts of this branch that still have a target: kept: - qwen3.5_397b_a17b_fp8 128,128 -> TIMEOUT(120) (now condition 9) - qwen3_235b_a22b_fp4 con:8 -> TIMEOUT(120) (now condition 11) - deepseek_r1_0528_fp8 x2 moved out of the >90G condition into a new >120G condition, renumbered 9b -> 10b to match the new index dropped as obsolete (tests no longer exist on main): - llama_v3.3_70b_instruct_fp8 20000,2000 TIMEOUT(180) - llama_v3.1_nemotron_ultra_253b 128,128 TIMEOUT(180) - llama_v3.3_70b_instruct_fp4 reqs:3000 TIMEOUT(120) - deepseek_v3.2_fp8 x2 (whole family removed upstream) Test count is unchanged vs main (164); no case added, removed or duplicated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-08-06T07:34:49Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

ruodil · 2026-08-06T07:41:06Z

/bot skip --comment "skip test as just adding cases"

tensorrt-cicd · 2026-08-06T07:47:35Z

PR_Github #64268 [ skip ] triggered by Bot. Commit: 0ff0b09 Link to invocation

tensorrt-cicd · 2026-08-06T08:01:37Z

PR_Github #64268 [ skip ] completed with state SUCCESS. Commit: 0ff0b09
Skipping testing for commit 0ff0b09

Link to invocation

ruodil added 2 commits August 3, 2026 05:55

ruodil requested a review from a team as a code owner August 3, 2026 06:18

ruodil requested review from LarryXFly and yufeiwu-nv August 3, 2026 06:18

github-actions Bot assigned ruodil Aug 3, 2026

coderabbitai Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread tests/integration/test_lists/qa/llm_perf_core.yml Outdated

Comment thread tests/integration/test_lists/qa/llm_perf_core.yml Outdated

BowenFu approved these changes Aug 3, 2026

View reviewed changes

Merge branch 'main' into user/ruodil/perf-core-timeout-slow-platforms

17187c0

ruodil enabled auto-merge (squash) August 3, 2026 09:10

yufeiwu-nv reviewed Aug 5, 2026

View reviewed changes

Merge branch 'main' into user/ruodil/perf-core-timeout-slow-platforms

0ff0b09

yufeiwu-nv approved these changes Aug 6, 2026

View reviewed changes

ruodil merged commit aafc4eb into NVIDIA:main Aug 6, 2026
7 checks passed

Navigation Menu

Uh oh!

[None][test] Fix perf-core timeouts and 96G-part coverage for large-checkpoint cases - #17183

[None][test] Fix perf-core timeouts and 96G-part coverage for large-checkpoint cases#17183
ruodil merged 5 commits into
NVIDIA:mainfrom
ruodil:user/ruodil/perf-core-timeout-slow-platforms

ruodil commented Aug 3, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 3, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

yufeiwu-nv left a comment

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

ruodil commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ruodil commented Aug 3, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yufeiwu-nv left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

ruodil commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ruodil commented Aug 3, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 3, 2026 •

edited

Loading