Skip to content

[[TRTLLM-13948][feat] Clean up DeepSeek tests using CPP Transceiver v1 #17090

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asfiyab-nvidia wants to merge 14 commits into
base: main
Choose a base branch
from
Open

[[TRTLLM-13948][feat] Clean up DeepSeek tests using CPP Transceiver v1 #17090

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
b7a6b3c
[[TRTLLM-13948][feat] Clean up DeepSeek tests using CPP Transceiver v1
asfiyab-nvidia Jul 30, 2026
84eb8ab
remove test_kv_cache_v2_nixl_python and keep test_auto_dtype for more…
asfiyab-nvidia Aug 3, 2026
f8381b0
remove TestDeepSeekV3Lite::test_auto_dtype_with_helix as helix can't …
asfiyab-nvidia Aug 3, 2026
5e40d82
Update DSV3-Lite to retain test_auto_dtype and remove test_kv_cache_v…
asfiyab-nvidia Aug 3, 2026
4fc0b7d
TestDeepSeekV3Lite::test_auto_dtype_with_helix to use CPP transceiver
asfiyab-nvidia Aug 3, 2026
903fc8b
minor rearrange
asfiyab-nvidia Aug 3, 2026
6164283
migrate deepseek configs to Python transceiver for tests/integration/…
asfiyab-nvidia Aug 4, 2026
0d1fdc4
remove disagg_config_ctxtp2_gentp2_deepseek_v3_lite_transceiver_runti…
asfiyab-nvidia Aug 4, 2026
b615d28
remove test_disaggregated_deepseek_v3_lite_fp8_transceiver_runtime_py…
asfiyab-nvidia Aug 4, 2026
d0dbb39
remove test_disaggregated_deepseek_v3_lite_fp8_ucx_tp1_single_gpu and…
asfiyab-nvidia Aug 4, 2026
6408eaf
Revert "remove test_disaggregated_deepseek_v3_lite_fp8_ucx_tp1_single…
asfiyab-nvidia Aug 4, 2026
5536469
Add config for TP1 UCX test
asfiyab-nvidia Aug 4, 2026
569c7c5
fix rebase bugs
asfiyab-nvidia Aug 5, 2026
e6e7f0b
fix rebase bugs
asfiyab-nvidia Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions tests/integration/defs/.test_durations
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
{
"accuracy/test_disaggregated_serving.py::TestDeepSeekR1::test_kv_cache_v2_nixl_python[cache_mgr_v1]": 563.5671428571428,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype[False]": 577.3825454545454,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_kv_cache_v2_nixl_python[cache_mgr_v1]": 762.6861428571428,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=0-overlap_scheduler=False]": 248.946875,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=0-overlap_scheduler=True]": 155.819,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=2-overlap_scheduler=False]": 226.137625,
Expand All @@ -10,13 +9,10 @@
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_gen_first[adp-mtp2]": 309.67263636363634,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_gen_first[noadp-mtp0]": 459.758,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_gen_only_spec_dec": 174.414125,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_gen_only_sync[python]": 261.8368118811881,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_guided_decoding[llguidance-mtp_nextn=0]": 164.20842857142858,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_guided_decoding[llguidance-mtp_nextn=2]": 373.866,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_guided_decoding[xgrammar-mtp_nextn=0]": 141.08328571428572,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_guided_decoding[xgrammar-mtp_nextn=2]": 176.04781981981984,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_kv_cache_v2_nixl_python": 283.61367924528304,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_nixl_backend": 256.39867676767676,
"accuracy/test_disaggregated_serving.py::TestDeepSeekV4Flash::test_auto_dtype": 653.3680224719101,
"accuracy/test_disaggregated_serving.py::TestGLM52NVFP4::test_nvfp4_nixl_python[cache_mgr_v1]": 1014.7549897959184,
"accuracy/test_disaggregated_serving.py::TestGPTOSS::test_kv_cache_v2_nixl_python[cache_mgr_v1]": 337.72714285714284,
Expand Down Expand Up @@ -650,7 +646,6 @@
"disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_bf16_empty_batch[DeepSeek-V3-Lite-bf16]": 190.4600707070707,
"disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_attention_dp[DeepSeek-V3-Lite-fp8]": 128.8671616161616,
"disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_nixl[DeepSeek-V3-Lite-fp8]": 121.09470707070707,
"disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_transceiver_runtime_python[DeepSeek-V3-Lite-fp8]": 118.99839393939394,
"disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_ucx[DeepSeek-V3-Lite-fp8]": 99.04030303030302,
"disaggregated/test_disaggregated.py::test_disaggregated_deepseek_v3_lite_fp8_ucx_tp1_single_gpu[DeepSeek-V3-Lite-fp8]": 136.5389226006192,
"disaggregated/test_disaggregated.py::test_disaggregated_diff_max_tokens[TinyLlama-1.1B-Chat-v1.0]": 72.53091237113402,
Expand Down
234 changes: 51 additions & 183 deletions tests/integration/defs/accuracy/test_disaggregated_serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1044,62 +1044,25 @@ class TestDeepSeekV3Lite(LlmapiAccuracyTestHarness):
@pytest.mark.skip_less_device(2)
@pytest.mark.skip_less_device_memory(60000)
@skip_no_hopper
def test_nixl_backend(self):
ctx_server_config = {
"disable_overlap_scheduler": True,
"cache_transceiver_config": {
"backend": "NIXL",
"max_tokens_in_buffer": 4096
}
}
gen_server_config = {
"disable_overlap_scheduler": True,
"cache_transceiver_config": {
"backend": "NIXL",
"max_tokens_in_buffer": 4096
}
}
disaggregated_server_config = {
"hostname": "localhost",
"backend": "pytorch",
"context_servers": {
"num_instances": 1
},
"generation_servers": {
"num_instances": 1
}
}
with launch_disaggregated_llm(disaggregated_server_config,
ctx_server_config, gen_server_config,
self.MODEL_PATH) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["MMLU", "GSM8K"])

@pytest.mark.skip_less_device(2)
@pytest.mark.skip_less_device_memory(60000)
@skip_no_hopper
@pytest.mark.parametrize("transceiver_runtime", ["PYTHON", "CPP"],
ids=["python", "cpp"])
def test_gen_only_sync(self, transceiver_runtime):
"""Test gen-only synchronous KV transfer with each NIXL runtime.
def test_gen_only_sync(self):

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add the required return annotation.

-def test_gen_only_sync(self):
+def test_gen_only_sync(self) -> None:

As per coding guidelines, every Python function must be annotated.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def test_gen_only_sync(self):
def test_gen_only_sync(self) -> None:
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/accuracy/test_disaggregated_serving.py` at line 1056,
Update the test_gen_only_sync method with the required return type annotation,
using the appropriate annotation for its test method behavior.

Source: Coding guidelines

"""Test gen-only synchronous KV transfer with PYTHON NIXL runtime.

Sets TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP=1 so the gen worker calls
the blocking request_and_receive_sync path. The C++ variant uses a
bounded client timeout so a stuck transfer fails this test instead of
waiting for its outer one-hour timeout.
the blocking request_and_receive_sync path.
"""
ctx_server_config = {
"disable_overlap_scheduler": True,
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": transceiver_runtime,
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096,
},
}
gen_server_config = {
"disable_overlap_scheduler": True,
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": transceiver_runtime,
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096,
},
}
Expand All @@ -1120,10 +1083,8 @@ def test_gen_only_sync(self, transceiver_runtime):
self.MODEL_PATH,
# Apply to both servers: gen worker uses sync receive path.
extra_env={"TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP": "1"},
request_timeout_s=(120 if transceiver_runtime == "CPP" else
DEFAULT_REQUEST_TIMEOUT_S),
request_max_retries=(0
if transceiver_runtime == "CPP" else None),
request_timeout_s=DEFAULT_REQUEST_TIMEOUT_S,
request_max_retries=None,
) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"])

Expand Down Expand Up @@ -1160,47 +1121,6 @@ def test_gen_only_spec_dec(self):
tensor_parallel_size=4) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["MMLU", "GSM8K"])

@pytest.mark.skip_less_device(8)
@parametrize_with_ids("overlap_scheduler", [True, False])
@parametrize_with_ids("mtp_nextn", [0, 2])
@pytest.mark.skip_less_device(8)
def test_auto_dtype(self, overlap_scheduler, mtp_nextn):
ctx_server_config = {"disable_overlap_scheduler": True}
gen_server_config = {"disable_overlap_scheduler": not overlap_scheduler}
ctx_server_config["cache_transceiver_config"] = {
"backend": "DEFAULT",
"max_tokens_in_buffer": 4096
}
gen_server_config["cache_transceiver_config"] = {
"backend": "DEFAULT",
"max_tokens_in_buffer": 4096
}
if mtp_nextn > 0:
ctx_server_config["speculative_config"] = {
"decoding_type": "MTP",
"max_draft_len": mtp_nextn
}
gen_server_config["speculative_config"] = {
"decoding_type": "MTP",
"max_draft_len": mtp_nextn
}
disaggregated_server_config = {
"hostname": "localhost",
"backend": "pytorch",
"context_servers": {
"num_instances": 1
},
"generation_servers": {
"num_instances": 1
}
}
with launch_disaggregated_llm(disaggregated_server_config,
ctx_server_config,
gen_server_config,
self.MODEL_PATH,
tensor_parallel_size=4) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["MMLU", "GSM8K"])

@skip_pre_blackwell
@pytest.mark.skip_less_device(8)
@pytest.mark.parametrize("gen_pp,gen_tp,gen_cp,enable_attention_dp", [
Expand Down Expand Up @@ -1286,33 +1206,22 @@ def test_auto_dtype_with_helix(self, comms_medium, cuda_graph_config,
self.MODEL_PATH) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["MMLU", "GSM8K"])

@pytest.mark.skip_less_device(2)
@pytest.mark.skip_less_device_memory(60000)
@pytest.mark.skip_less_device(8)
@parametrize_with_ids("overlap_scheduler", [True, False])
@parametrize_with_ids("mtp_nextn", [0, 2])
@pytest.mark.parametrize("backend", ["xgrammar", "llguidance"])
def test_guided_decoding(self, backend: str, mtp_nextn: int, mocker):
mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"})
ctx_server_config = {
"disable_overlap_scheduler": True,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.8,
},
"guided_decoding_backend": backend,
"cache_transceiver_config": {
"backend": "DEFAULT",
"max_tokens_in_buffer": 4096
}
@pytest.mark.skip_less_device(8)
def test_auto_dtype(self, overlap_scheduler, mtp_nextn):
ctx_server_config = {"disable_overlap_scheduler": True}
gen_server_config = {"disable_overlap_scheduler": not overlap_scheduler}
ctx_server_config["cache_transceiver_config"] = {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
}
gen_server_config = {
"disable_overlap_scheduler": False,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.8,
},
"guided_decoding_backend": backend,
"cache_transceiver_config": {
"backend": "DEFAULT",
"max_tokens_in_buffer": 4096
}
gen_server_config["cache_transceiver_config"] = {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
}
if mtp_nextn > 0:
ctx_server_config["speculative_config"] = {
Expand All @@ -1334,37 +1243,51 @@ def test_guided_decoding(self, backend: str, mtp_nextn: int, mocker):
}
}
with launch_disaggregated_llm(disaggregated_server_config,
ctx_server_config, gen_server_config,
self.MODEL_PATH) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["JsonModeEval"])
ctx_server_config,
gen_server_config,
self.MODEL_PATH,
tensor_parallel_size=4) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["MMLU", "GSM8K"])

@pytest.mark.skip_less_device(2)
@pytest.mark.skip_less_device_memory(60000)
@skip_pre_hopper
def test_kv_cache_v2_nixl_python(self):
"""Test with use_kv_cache_manager_v2=True, block_reuse=False, backend=NIXL, transceiver_runtime=PYTHON."""
@parametrize_with_ids("mtp_nextn", [0, 2])
@pytest.mark.parametrize("backend", ["xgrammar", "llguidance"])
def test_guided_decoding(self, backend: str, mtp_nextn: int, mocker):
mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"})
ctx_server_config = {
"disable_overlap_scheduler": True,
"kv_cache_config": {
"enable_block_reuse": False,
"use_kv_cache_manager_v2": True
"free_gpu_memory_fraction": 0.8,
},
"guided_decoding_backend": backend,
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON"
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
}
}
gen_server_config = {
"disable_overlap_scheduler": True,
"disable_overlap_scheduler": False,
"kv_cache_config": {
"enable_block_reuse": False,
"use_kv_cache_manager_v2": True
"free_gpu_memory_fraction": 0.8,
},
"guided_decoding_backend": backend,
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON"
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
}
}
if mtp_nextn > 0:
ctx_server_config["speculative_config"] = {
"decoding_type": "MTP",
"max_draft_len": mtp_nextn
}
gen_server_config["speculative_config"] = {
"decoding_type": "MTP",
"max_draft_len": mtp_nextn
}
disaggregated_server_config = {
"hostname": "localhost",
"backend": "pytorch",
Expand All @@ -1378,7 +1301,7 @@ def test_kv_cache_v2_nixl_python(self):
with launch_disaggregated_llm(disaggregated_server_config,
ctx_server_config, gen_server_config,
self.MODEL_PATH) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"])
run_accuracy_test(llm, self.MODEL_NAME, ["JsonModeEval"])

@pytest.mark.skip_less_device(4)
@pytest.mark.skip_less_device_memory(60000)
Expand Down Expand Up @@ -1680,7 +1603,8 @@ class TestDeepSeekV32Exp(LlmapiAccuracyTestHarness):
@pytest.mark.parametrize("overlap_scheduler", [False])
def test_auto_dtype(self, overlap_scheduler):
cache_transceiver_config = {
"backend": "DEFAULT",
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
}
max_num_tokens = 8192
Expand Down Expand Up @@ -1740,62 +1664,6 @@ def test_auto_dtype(self, overlap_scheduler):
model_name=self.MODEL_NAME,
test_sets=["MMLU", "GSM8K"])

@pytest.mark.skip_less_device(4)
@pytest.mark.skip_less_device_memory(200000)
@pytest.mark.parametrize("use_kv_cache_manager_v2", [False],
ids=["cache_mgr_v1"])
def test_kv_cache_v2_nixl_python(self, use_kv_cache_manager_v2):
"""Test with KV cache manager v1, block_reuse=False, backend=NIXL, transceiver_runtime=PYTHON."""
max_num_tokens = 8192
moe_config = {"backend": "TRTLLM", "max_num_tokens": max_num_tokens}
ctx_server_config = {
"disable_overlap_scheduler": True,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.5,
"enable_block_reuse": False,
"use_kv_cache_manager_v2": use_kv_cache_manager_v2
},
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
},
"tensor_parallel_size": 2,
"moe_expert_parallel_size": 2,
"enable_autotuner": False,
}
gen_server_config = {
"disable_overlap_scheduler": False,
"moe_config": moe_config,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.5,
"enable_block_reuse": False,
"use_kv_cache_manager_v2": use_kv_cache_manager_v2
},
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
},
"tensor_parallel_size": 2,
"moe_expert_parallel_size": 2,
"enable_autotuner": False,
}
disaggregated_server_config = {
"hostname": "localhost",
"backend": "pytorch",
"context_servers": {
"num_instances": 1,
},
"generation_servers": {
"num_instances": 1,
}
}
with launch_disaggregated_llm(disaggregated_server_config,
ctx_server_config, gen_server_config,
self.MODEL_PATH) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"])


@pytest.mark.timeout(DEFAULT_TEST_TIMEOUT)
@skip_pre_hopper
Expand Down
6 changes: 4 additions & 2 deletions ...ration/defs/disaggregated/test_configs/disagg_config_cache_aware_balance_deepseek_v3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ context_servers:
event_buffer_max_size: 1024
free_gpu_memory_fraction: 0.1
cache_transceiver_config:
backend: DEFAULT
backend: NIXL
transceiver_runtime: PYTHON
generation_servers:
num_instances: 2
router:
Expand All @@ -29,4 +30,5 @@ generation_servers:
event_buffer_max_size: 1024
free_gpu_memory_fraction: 0.1
cache_transceiver_config:
backend: DEFAULT
backend: NIXL
transceiver_runtime: PYTHON
6 changes: 4 additions & 2 deletions tests/integration/defs/disaggregated/test_configs/disagg_config_cache_reuse_deepseek_v3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ context_servers:
enable_partial_reuse: true
event_buffer_max_size: 1024
cache_transceiver_config:
backend: DEFAULT
backend: NIXL
transceiver_runtime: PYTHON
generation_servers:
num_instances: 1
tensor_parallel_size: 1
Expand All @@ -30,4 +31,5 @@ generation_servers:
event_buffer_max_size: 1024
free_gpu_memory_fraction: 0.05
cache_transceiver_config:
backend: DEFAULT
backend: NIXL
transceiver_runtime: PYTHON
Loading
Loading