Skip to content

[https://nvbugs/6435112][test] Unwaive Wan 2.2 I2V perf sanity test#16409

Open
taianz-nv wants to merge 1 commit into
NVIDIA:mainfrom
taianz-nv:dev-taianz-unwaive-6435112
Open

[https://nvbugs/6435112][test] Unwaive Wan 2.2 I2V perf sanity test#16409
taianz-nv wants to merge 1 commit into
NVIDIA:mainfrom
taianz-nv:dev-taianz-unwaive-6435112

Conversation

@taianz-nv

@taianz-nv taianz-nv commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Remove the waiver for the Wan 2.2 I2V A14B NVFP4 TRT-LLM Ulysses-4 VisualGen perf-sanity test.
  • Keep the test enabled in the B200 VisualGen perf-sanity test list.
  • Leave all other waivers associated with NVBug 6422339 unchanged.

Background

NVBug 6435112 tracks an NCCL ALLTOALL_BASE watchdog timeout in this exact test. The repair bot reran the test multiple times on B200 x8 against current main and all attempts passed, so this PR re-enables the test for full CI validation.

NVBug context

This test was initially waived under NVBug 6422339, which groups multiple failures from Post-Merge #2824. NVBug 6435112 tracks this specific Wan 2.2 I2V NCCL timeout, so this PR is titled under 6435112 while removing the existing waiver linked to 6422339. Other 6422339 waivers are unchanged.

Validation

  • pre-commit run --files tests/integration/test_lists/waives.txt

CI

Please run DGX_B200-8_GPUs-PyTorch-VisualGen-PerfSanity-Post-Merge-1 to validate the previously flaky distributed test before closing NVBug 6435112.

Summary by CodeRabbit

  • Tests
    • Removed an obsolete skip waiver for a visual generation performance sanity test, allowing the test case to run normally.

Signed-off-by: Taian Zhang <taianz@nvidia.com>
@taianz-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-8_GPUs-PyTorch-VisualGen-PerfSanity-Post-Merge-1"

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Removes one SKIP waiver for a specific visual generation end-to-end performance test parameterization from the integration waiver list.

Changes

Visual generation waiver

Layer / File(s) Summary
Remove visual generation SKIP waiver
tests/integration/test_lists/waives.txt
Deletes the waiver for the targeted test_visual_gen_e2e parameterization.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: bowenfu, brnguyen2

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly matches the main change: removing a waiver for the Wan 2.2 I2V perf sanity test.
Description check ✅ Passed The description explains the change, background, validation, and CI follow-up, though the template headings are not exact.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59383 [ run ] triggered by Bot. Commit: d75d5f8 Link to invocation

@taianz-nv taianz-nv changed the title [nvbug/6435112] test: Unwaive Wan 2.2 I2V perf sanity [https://nvbugs/6435112][test] Unwaive Wan 2.2 I2V perf sanity test Jul 15, 2026
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59383 [ run ] completed with state SUCCESS. Commit: d75d5f8
/LLM/main/L0_MergeRequest_PR pipeline #47857 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@taianz-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-8_GPUs-PyTorch-VisualGen-PerfSanity-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59446 [ run ] triggered by Bot. Commit: d75d5f8 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #59446 [ run ] completed with state SUCCESS. Commit: d75d5f8
/LLM/main/L0_MergeRequest_PR pipeline #47914 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@fredricz-20070104 fredricz-20070104 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@fredricz-20070104

Copy link
Copy Markdown
Collaborator

Quick question to confirm the bug attribution before this merges.

The line removed in this PR is tests/integration/test_lists/waives.txt line 433 (diff hunk @@ -430,7 +430,6 @@):

-perf/test_visual_gen_perf_sanity.py::test_visual_gen_e2e[vg_upload-wan22_i2v_a14b_blackwell-wan22_i2v_a14b_nvfp4_trtllm_cfg2_ulysses4] SKIP (https://nvbugs/6422339)

That waiver is tagged nvbugs/6422339, but the PR title/description says this un-waives the case tracked by nvbugs/6435112 (NCCL ALLTOALL_BASE watchdog timeout). Could you confirm which bug actually caused this specific case to be skipped?

If the case was originally waived under 6422339 (which covers several perf-sanity cases) and only the 6435112 NCCL timeout was fixed, removing this waiver may re-expose the 6422339 failure. If 6422339 is in fact resolved for this case, it would help to update the description (or the waiver's bug reference) so the attribution is consistent. Thanks!

@taianz-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-8_GPUs-PyTorch-VisualGen-PerfSanity-Post-Merge-1"

@taianz-nv

Copy link
Copy Markdown
Collaborator Author

As discussed in DM, 6422339 is the Post-Merge aggregation bug that originally carried this waiver, while 6435112 specifically tracks the Wan 2.2 NCCL timeout. I have also updated the PR description to clarify this. Thanks!

@yufeiwu-nv yufeiwu-nv removed their request for review July 16, 2026 06:46

@BowenFu BowenFu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — single waiver removal (nvbugs/6435112, Wan 2.2 I2V perf sanity).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants