[None][test] Fix qwen3.5_9b avg_seq_len and restore H100 coverage for three Qwen3.5 perf cases - #17422
[None][test] Fix qwen3.5_9b avg_seq_len and restore H100 coverage for three Qwen3.5 perf cases#17422yufeiwu-nv wants to merge 3 commits into
Conversation
Qwen3.5-9B is a hybrid GDN model, so KV cache manager v2 splits memory between an attention pool (per-token cost) and an SSM/conv state pool (fixed cost per resident sequence). Without kv_cache_config.avg_seq_len it warns and falls back to max_seq_len/2, which underestimates the real sequence length by 2x and undersizes the attention pool relative to the SSM pool. On L40S the 500/2000 case then admits 485 resident sequences, saturates the attention pool at 99% utilization and aborts with a V2 scheduler deadlock. The synthetic dataset uses stdev 0, so 500+2000=2500 is the exact total sequence length rather than an estimate. Scoped to the failing case only; the other qwen3.5_9b cases currently pass and changing their pool ratio would shift their perf baselines. Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>
WalkthroughAdds a Qwen3.5-9B PyTorch KV-cache configuration and updates Qwen3.5 performance benchmark assignments across GPU test groups. ChangesQwen3.5 performance benchmark updates
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/integration/defs/perf/pytorch_model_config.py (1)
327-337: 🚀 Performance & Scalability | 🔵 TrivialTest coverage summary: sufficient.
- Test functions added, modified, or removed: none.
- Existing QA coverage:
tests/integration/test_lists/qa/llm_perf_core.ymlincludes the affectedinput_output_len:500,2000benchmark.- Coverage verdict: sufficient for this configuration-only change.
Run the affected integration case with GPU access and model weights. Set
LLM_MODELS_ROOTbefore running it.As per path instructions, integration tests require GPU access and model weights; set
LLM_MODELS_ROOTbefore running them.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/defs/perf/pytorch_model_config.py` around lines 327 - 337, Validate the added Qwen3.5-9B configuration by running the affected integration benchmark matching input_output_len:500,2000 with GPU access and model weights available; set LLM_MODELS_ROOT before execution. No code changes are required.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/defs/perf/pytorch_model_config.py`:
- Around line 329-330: Update the pattern in the PyTorch benchmark configuration
used by get_model_yaml_config so matching cannot accept values extending beyond
input_output_len:500,2000. Require a hyphen or end-of-label boundary after 2000,
or otherwise match the parsed fields exactly, and add a regression test ensuring
input_output_len:500,20000 is not selected.
---
Nitpick comments:
In `@tests/integration/defs/perf/pytorch_model_config.py`:
- Around line 327-337: Validate the added Qwen3.5-9B configuration by running
the affected integration benchmark matching input_output_len:500,2000 with GPU
access and model weights available; set LLM_MODELS_ROOT before execution. No
code changes are required.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1e669d50-d076-4535-9cac-c1d62bae60e7
📒 Files selected for processing (1)
tests/integration/defs/perf/pytorch_model_config.py
|
/bot run |
|
PR_Github #64628 [ run ] triggered by Bot. Commit: |
|
PR_Github #64628 [ run ] completed with state
|
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
Approving — the comments below are optional touch-ups, not blockers.
Config value and pattern both check out — the pattern matches the label built in test_perf.py:549, and matching is substring-based so the full label anchors it to this case only.
The description says the engine still spins 8689 iterations with zero forward progress after saturation and reports a missing host cache tier that was actually provisioned. That's a real bug this change only routes around. Please file an NVBug for it and reference it in a code comment here, otherwise the workaround loses its context once someone revisits the pool ratio for the other four cases.
yufeiwu-nv
changed the title
|
/bot run |
NVIDIA#17303 moved these two cases out of the H100-capable conditions: qwen3.5_122b_a10b input_output_len:500,2000 -ep:4-tp:4-gpus:4 qwen3.5_397b_a17b_fp8 input_output_len:1000,2000 -ep:8-tp:8-gpus:8 The first went to the gpu_memory > 90000 condition under an "OOM on 80G GPUs" rationale, the second to a B200/B300-only condition. The perf result database contradicts both: 500,2000 has 26 successful H100 runs (last 2026-07-30) out of 191 total, and 1000,2000 has 11 H100 runs (last 2026-07-30) out of 67 total. Move each back next to its siblings, which never left the H100 conditions. qwen3.5_122b_a10b input_output_len:128,128 stays where it is — it has 115 runs but none on H100, so it has no H100 baseline to restore. Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_lists/qa/llm_perf_core.yml`:
- Line 109: Add the standard NVIDIA copyright header to the top of the QA list
file, including year 2026, while preserving the existing test-list entry and
formatting.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d5ca54fd-7244-42f1-bf83-ac15d2a86f6d
📒 Files selected for processing (1)
tests/integration/test_lists/qa/llm_perf_core.yml
| - perf/test_perf.py::test_perf[qwen3.5_35b_a3b_fp8-bench-pytorch-float8-maxbs:1-input_output_len:8000,1000-reqs:10-con:1] #min_latency | ||
| - perf/test_perf.py::test_perf[qwen3.5_35b_a3b_fp8-bench-pytorch-float8-maxbs:512-input_output_len:8000,1000-con:256] #max_throughput | ||
| #qwen3.5_122b_a10b (MoE BF16 234G, 4-GPU) | ||
| - perf/test_perf.py::test_perf[qwen3.5_122b_a10b-bench-pytorch-bfloat16-input_output_len:500,2000-ep:4-tp:4-gpus:4] |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="tests/integration/test_lists/qa/llm_perf_core.yml"
sed -n '1,12p' "$file"
rg -n -i 'copyright|nvidia' "$file" || true
git diff --unified=0 -- "$file"Repository: NVIDIA/TensorRT-LLM
Length of output: 708
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="tests/integration/test_lists/qa/llm_perf_core.yml"
printf '%s\n' '--- file header ---'
sed -n '1,20p' "$file"
printf '%s\n' '--- copyright references in the file ---'
rg -n -i 'copyright|nvidia' "$file" || true
printf '%s\n' '--- change summary ---'
git diff --stat -- "$file"
printf '%s\n' '--- changed lines ---'
git diff --unified=3 -- "$file"
printf '%s\n' '--- comparable QA list headers ---'
for candidate in tests/integration/test_lists/qa/*.yml; do
if [ "$candidate" != "$file" ]; then
printf '%s\n' "### $candidate"
sed -n '1,8p' "$candidate"
break
fi
doneRepository: NVIDIA/TensorRT-LLM
Length of output: 1748
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- working-tree changes ---'
git status --short
git diff --name-status
git diff --cached --name-status
printf '%s\n' '--- target entry in QA lists ---'
rg -n -F 'perf/test_perf.py::test_perf[qwen3.5_122b_a10b-bench-pytorch-bfloat16-input_output_len:500,2000-ep:4-tp:4-gpus:4]' tests/integration/test_lists/qa tests/integration/test_lists/test-db || true
printf '%s\n' '--- CBTS artifacts ---'
find . -maxdepth 5 \( -name 'cbts_touchmap.sqlite' -o -iname '*cbts*coverage*' \) -printRepository: NVIDIA/TensorRT-LLM
Length of output: 414
Add the NVIDIA copyright header with year 2026. tests/integration/test_lists/qa/llm_perf_core.yml has no NVIDIA copyright header.
Test-list coverage: The modified QA list and added or removed entries cannot be determined. No CBTS touch map or coverage report is available. Verdict: needs follow-up.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/integration/test_lists/qa/llm_perf_core.yml` at line 109, Add the
standard NVIDIA copyright header to the top of the QA list file, including year
2026, while preserving the existing test-list entry and formatting.
Source: Coding guidelines
Follow-up to the previous commit, which left this case in the gpu_memory > 90000 condition on the grounds that it had no H100 history. A closer look at the perf result database shows the case was never executed on H100 at all, rather than executed and failed: it has zero perf_result rows on H100, and across the last 15 H100 runs that ran qwen3.5_122b_a10b input_output_len:500,2000 the 128,128 case is absent from every one. Its 115 runs on other GPUs all produced valid throughput, with no failures recorded anywhere. So the "OOM on 80G GPUs" rationale has no supporting data for this case either. 128,128 has a 256-token sequence against 2500 for the 500,2000 case that does pass on H100, so it is the lighter workload of the two. Move it back next to its siblings and drop the now-empty comment. Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>
yufeiwu-nv
changed the title
|
/bot run |
|
PR_Github #64691 [ run ] triggered by Bot. Commit: |
|
PR_Github #64691 [ run ] completed with state
|
3 participants
Change 1: avg_seq_len for the qwen3.5_9b 500/2000 case
Background
perf/test_perf.py::test_perf[qwen3.5_9b-bench-pytorch-bfloat16-input_output_len:500,2000]fails on L40S with:Qwen3.5-9B is a hybrid Gated DeltaNet model, so it runs on
MambaHybridCacheManagerV2, which splits GPU memory into an attention pool (per-token cost) and an SSM/conv state pool (fixed cost per resident sequence). The split comes from_get_typical_request_capacity(), which warns and falls back tomax_seq_len / 2whenkv_cache_config.avg_seq_lenis unset:For this case the real total sequence length is 2500, so the fallback underestimates it by 2x, undersizes the attention pool relative to the SSM pool, and lets the scheduler admit more resident sequences than the attention pool can sustain.
Summary
Set
kv_cache_config.avg_seq_len = 2500for this one perf case. The synthetic dataset is generated with--input-stdev=0 --output-stdev=0, so 500+2000 is the exact total sequence length, not an estimate.Impact
Scoped to the failing case only. The other four
qwen3.5_9bcases use the same fallback but currently pass, and changing their pool ratio would shift their perf baselines — that belongs in a separate change.Evidence that peak concurrency is the discriminator (same GPU, same 22.44 GiB KV quota, same build):
Setting
avg_seq_lenshifts memory toward the attention pool and reduces the number of SSM slots, so this case should settle at a concurrency the attention pool can actually sustain.Note this only addresses the pool-sizing mismatch. The underlying engine behaviour still looks wrong: after saturation the executor ran 8689 iterations with zero forward progress (0.9 ms/iter, no forward pass) while GPU KV utilization climbed from 0.019 to 0.898, and the deadlock message blames a missing host cache tier even though one was auto-provisioned at 22.44 GiB in the same run.
Test plan
perf/test_perf.py::test_perf[qwen3.5_9b-bench-pytorch-bfloat16-input_output_len:500,2000]passes on L40Sqwen3.5_9bperf cases are unaffected (pattern is scoped to the full test label)Change 2: restore H100 coverage for two Qwen3.5 cases
Background
#17303 moved two cases out of the H100-capable conditions in
llm_perf_core.yml:qwen3.5_122b_a10b ... input_output_len:500,2000-ep:4-tp:4-gpus:4gpu_memory > 90000OOM on 80G GPUs, needs >90Gqwen3.5_397b_a17b_fp8 ... input_output_len:1000,2000-ep:8-tp:8-gpus:8H100 is 80GB, so the first condition excludes it.
Why this was wrong
The perf result database contradicts both moves:
qwen3.5_122b_a10b500,2000qwen3.5_397b_a17b_fp81000,2000Both have a substantial H100 history, so the OOM rationale does not hold for them.
Summary
Each case moves back next to its siblings, which never left the H100 conditions:
qwen3.5_122b_a10b500,2000returns to theH100, H20, GB200, B200, B300, GB300, RTX6000-Servercondition, joining2000,500,8000,1000and1000,2000.qwen3.5_397b_a17b_fp81000,2000returns to theH100, H20, B200, B300, RTX6000-Servercondition, joining128,128,500,2000,2000,500and8000,1000.qwen3.5_122b_a10binput_output_len:128,128moves back as well. It has zeroperf_resultrows on H100, but that is because it was never executed there, not because it failed: across the last 15 H100 runs that executed the500,2000case,128,128is absent from every one. Its 115 runs on other GPUs all produced valid throughput with no failures recorded anywhere, and at a 256-token sequence it is the lighter of the two workloads, so theOOM on 80G GPUsrationale has no supporting data for it either. The now-empty comment is dropped.Test plan
check_test_list.py --validatepasses (2060 entries)llm_perf_core.yml)