Skip to content

[https://nvbugs/6611817][test] Waive known-flaky disagg + Nemotron tests on DGX_H100 - #17712

Merged
brnguyen2 merged 1 commit into
NVIDIA:mainfrom
brnguyen2:test/waive-disagg-nemotron-flakes
Aug 14, 2026
Merged

[https://nvbugs/6611817][test] Waive known-flaky disagg + Nemotron tests on DGX_H100#17712
brnguyen2 merged 1 commit into
NVIDIA:mainfrom
brnguyen2:test/waive-disagg-nemotron-flakes

Conversation

@brnguyen2

@brnguyen2 brnguyen2 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Description

Standalone waives-only PR: SKIP-waive three tests that flake on main CI,
each with a real tracking bug. These were pulled out of #17482 so they land
fast with real bug references instead of the placeholder-then-amend flow.

All three flake on stage DGX_H100-4_GPUs-PyTorch-Others-2, independent of any
in-flight branch (they fail across 110+ unrelated MRs and multiple users on main):

Test Flake (14d) Bug
test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp2pp1] 1.3% / 113 MRs nvbugs/6611817 (new)
test_auto_scaling.py::test_disagg_server_restart[etcd-round_robin] 1.2% / 112 MRs nvbugs/6611817 (shared)
test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_4gpus_hopper_w4a16 flaky nvbugs/6478723 (existing, TRTLLM-14447)

The first two co-fail on the same stage in the same runs — a shared disagg
server timeout/hang — so they share one new tracking bug (6611817). The
Nemotron nvfp4 Hopper flake folds into its existing open tracking bug.

Test Coverage

waives-only change (no source). Validated by the pre-commit hooks that ran on
commit: "Validate test list entries exist in source files (AST)", "Checks for
duplicated test items in waives.txt", and the sort/format checks — all passed.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • Added three SKIP waivers to tests/integration/test_lists/waives.txt.
  • The waivers cover two disaggregated serving tests linked to nvbugs/6611817.
  • The Nemotron nvFP4 Hopper test is linked to nvbugs/6478723 (TRTLLM-14447).
  • No source or public API changes were made.
  • Pre-commit validation confirmed valid formatting, sorting, test entries, and no duplicates.

Verdict: sufficient.

QA Engineer Review

  • Only tests/integration/test_lists/waives.txt changed.
  • Added three waiver entries:
    • Disaggregated-serving Llama 3.1 symmetric MMLU test.
    • PyTorch Nemotron V3 Super NVFP4 Hopper test.
    • Disaggregated server restart test with etcd and round-robin scaling.
  • No test-db/ or qa/ files were modified.
  • CBTS coverage data is unavailable.

Verdict: needs follow-up.

…sts on DGX_H100

Three tests flake on main CI (stage DGX_H100-4_GPUs-PyTorch-Others-2),
independent of any branch, so SKIP-waive them with their tracking bugs:

- accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp2pp1]
    disagg-server timeout/hang, ~1.3% over 113 MRs -> nvbugs/6611817
- disaggregated/test_auto_scaling.py::test_disagg_server_restart[etcd-round_robin]
    co-fails on the same stage/runs, ~1.2% over 112 MRs -> nvbugs/6611817 (shared)
- accuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_4gpus_hopper_w4a16
    nvfp4 Hopper w4a16 flake -> nvbugs/6478723 (existing, TRTLLM-14447)

Pulled out of PR NVIDIA#17482 into a standalone waives-only PR so they land fast
with real bug references instead of placeholders.

Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4b6bda02-4cba-48fa-bda0-f45ad4d815b7

📥 Commits

Reviewing files that changed from the base of the PR and between 7a3b1bf and 4d406ed.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt

Walkthrough

Three integration tests are added to the waiver list. Each entry references NVBug 6611817.

Changes

Integration test waivers

Layer / File(s) Summary
Add skipped integration test entries
tests/integration/test_lists/waives.txt
Adds skip entries for the Llama 3.1 symmetric MMLU test, the Nemotron V3 Super NVFP4 Hopper test, and the etcd round-robin disaggregated server restart test.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 4d406

This PR only adds three known-flaky test waivers and does not change product or runtime code; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: waiving known-flaky disaggregated serving and Nemotron tests on DGX_H100.
Description check ✅ Passed The description explains the issue, lists all three waived tests and tracking bugs, documents validation, and includes the required checklist.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@brnguyen2
brnguyen2 enabled auto-merge (squash) August 14, 2026 17:00
@brnguyen2

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "waives only PR"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66322 [ skip ] triggered by Bot. Commit: 4d406ed Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #66322 [ skip ] completed with state SUCCESS. Commit: 4d406ed
Skipping testing for commit 4d406ed

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants