Skip to content

[TRTLLM-13948][test] Add DeepSeek R1/V3.2/V3-Lite disaggregated accur… #16344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asfiyab-nvidia wants to merge 3 commits into
base: main
Choose a base branch
from
+124 −5
Open

[TRTLLM-13948][test] Add DeepSeek R1/V3.2/V3-Lite disaggregated accur… #16344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
648b5a5
[TRTLLM-13948][test] Add DeepSeek R1/V3.2/V3-Lite disaggregated accur…
asfiyab-nvidia Jul 13, 2026
6e02fbd
changes:
asfiyab-nvidia Jul 14, 2026
d555a53
disable autotuner for DeepSeekR1 test
asfiyab-nvidia Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 120 additions & 5 deletions tests/integration/defs/accuracy/test_disaggregated_serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1371,15 +1371,12 @@ def test_kv_cache_v2_nixl_python(self):
}
disaggregated_server_config = {
"hostname": "localhost",
"port": 8000,
"backend": "pytorch",
"context_servers": {
"num_instances": 1,
"urls": ["localhost:8001"]
"num_instances": 1
},
"generation_servers": {
"num_instances": 1,
"urls": ["localhost:8002"]
"num_instances": 1
}
}
with launch_disaggregated_llm(disaggregated_server_config,
Expand Down Expand Up @@ -1843,6 +1840,61 @@ def test_auto_dtype_with_helix(self, comms_medium, cuda_graph_config,
max_workers=128) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"])

@pytest.mark.skip_less_device(4)
@pytest.mark.parametrize("use_kv_cache_manager_v2", [False],
ids=["cache_mgr_v1"])
def test_kv_cache_v2_nixl_python(self, use_kv_cache_manager_v2):

@chienchunhung chienchunhung Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This test does not have memory guard.

"""Test with KV cache manager v1, block_reuse=False, backend=NIXL, transceiver_runtime=PYTHON."""
max_num_tokens = 8192
moe_config = {"backend": "TRTLLM", "max_num_tokens": max_num_tokens}
ctx_server_config = {
"disable_overlap_scheduler": True,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.5,
"enable_block_reuse": False,
"use_kv_cache_manager_v2": use_kv_cache_manager_v2
},
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
},
"tensor_parallel_size": 2,
"moe_expert_parallel_size": 2,
"enable_autotuner": False,
}
gen_server_config = {
"disable_overlap_scheduler": False,
"moe_config": moe_config,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.5,
"enable_block_reuse": False,
"use_kv_cache_manager_v2": use_kv_cache_manager_v2
},
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
},
"tensor_parallel_size": 2,
"moe_expert_parallel_size": 2,
"enable_autotuner": False,
}
disaggregated_server_config = {
"hostname": "localhost",
"backend": "pytorch",
"context_servers": {
"num_instances": 1,
},
"generation_servers": {
"num_instances": 1,
}
}
with launch_disaggregated_llm(disaggregated_server_config,
ctx_server_config, gen_server_config,
self.MODEL_PATH) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"])


@pytest.mark.timeout(DEFAULT_TEST_TIMEOUT)
@skip_pre_hopper
Expand Down Expand Up @@ -2754,3 +2806,66 @@ def test_auto_dtype(self):
server_waiting_timeout=3600) as llm:
task = MMLU(self.MODEL_NAME)
task.evaluate(llm, is_integration_test=True)


@pytest.mark.timeout(DEFAULT_TEST_TIMEOUT)
@skip_pre_blackwell
@pytest.mark.skip_less_device_memory(80000)
class TestDeepSeekR1(LlmapiAccuracyTestHarness):
MODEL_NAME = "deepseek-ai/DeepSeek-R1"
MODEL_PATH = f"{llm_models_root()}/DeepSeek-R1-0528-FP4-v2"

@nv-xtf nv-xtf Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please verify the R1 model path? The existing CI model mapping and R1 tests use DeepSeek-R1/DeepSeek-R1-0528-FP4-v2, while this test currently omits the DeepSeek-R1/ parent directory. Unless there is a separate top-level symlink, this is likely to fail when loading the model.

@chienchunhung chienchunhung Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be f"{llm_models_root()}/DeepSeek-R1/DeepSeek-R1-0528-FP4-v2"?


@pytest.mark.skip_less_device(4)
@pytest.mark.parametrize("use_kv_cache_manager_v2", [False],
ids=["cache_mgr_v1"])
def test_kv_cache_v2_nixl_python(self, use_kv_cache_manager_v2):
"""Test with KV cache manager v1, block_reuse=False, backend=NIXL, transceiver_runtime=PYTHON."""
max_num_tokens = 8192
moe_config = {"backend": "TRTLLM", "max_num_tokens": max_num_tokens}
ctx_server_config = {
"disable_overlap_scheduler": True,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.5,
"enable_block_reuse": False,
"use_kv_cache_manager_v2": use_kv_cache_manager_v2
},
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
},
"tensor_parallel_size": 2,
"moe_expert_parallel_size": 2,
"enable_autotuner": False,
}
gen_server_config = {
"disable_overlap_scheduler": False,
"moe_config": moe_config,
"kv_cache_config": {
"free_gpu_memory_fraction": 0.5,
"enable_block_reuse": False,
"use_kv_cache_manager_v2": use_kv_cache_manager_v2
},
"cache_transceiver_config": {
"backend": "NIXL",
"transceiver_runtime": "PYTHON",
"max_tokens_in_buffer": 4096
},
"tensor_parallel_size": 2,
"moe_expert_parallel_size": 2,
"enable_autotuner": False,
}
disaggregated_server_config = {
"hostname": "localhost",
"backend": "pytorch",
"context_servers": {
"num_instances": 1,
},
"generation_servers": {
"num_instances": 1,
}
}
with launch_disaggregated_llm(disaggregated_server_config,
ctx_server_config, gen_server_config,
self.MODEL_PATH) as llm:
run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"])
Comment thread
coderabbitai[bot] marked this conversation as resolved.
2 changes: 2 additions & 0 deletions tests/integration/test_lists/qa/llm_function_core.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
accuracy/test_disaggregated_serving.py::TestDeepSeekR1::test_kv_cache_v2_nixl_python[cache_mgr_v1]
accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype[False]
accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype_with_helix[fifo-cudagraph:with_padding-pp1dp2cp2]
accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype_with_helix[fifo-cudagraph:with_padding-pp1tp1cp4]
accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype_with_helix[fifo-cudagraph:with_padding-pp1tp2cp2]
accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype_with_helix[fifo-cudagraph:with_padding-pp2tp1cp2]
accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_kv_cache_v2_nixl_python[cache_mgr_v1]
accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=0-overlap_scheduler=False]
accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=0-overlap_scheduler=True]
accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=2-overlap_scheduler=False]
Expand Down
2 changes: 2 additions & 0 deletions tests/integration/test_lists/test-db/l0_dgx_b300.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ l0_dgx_b300:
- accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4a16[dp4-auto]
- accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4a16[dp4-fp8]
- accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_nixl_backend
- accuracy/test_disaggregated_serving.py::TestDeepSeekR1::test_kv_cache_v2_nixl_python[cache_mgr_v1]

@nv-xtf nv-xtf Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the new accuracy test IDs are added under the stage: post_merge condition. If the intention is to use them as the PR pre-merge gate for the migration, they will not run automatically in the standard pre-merge pipeline. Is post-merge placement intentional, with targeted bot-run validation planned?

- accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_kv_cache_v2_nixl_python[cache_mgr_v1]
- accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_nixl_backend
- accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_kv_cache_v2_nixl_python
- accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_kv_cache_v2_nixl_python
Expand Down
Loading