fix: resolve SessionWatcher goroutine leak and unstable UT in querycoordv2 #45627

congqixia · 2025-11-17T11:00:42Z

Related to #44620
Related to unstable ut "internal/querycoordv2 TestServer/TestNodeUp"

Introduce SessionWatcher interface to fix race condition and goroutine leak that caused unstable unit test TestServer/TestNodeUp.

Changes:

Add SessionWatcher interface with EventChannel() and Stop() methods
Refactor WatchServices() to return SessionWatcher instead of raw channel
Fix cleanup order in QueryCoordV2: stop watcher before session
Update DataCoord, ConnectionManager to use SessionWatcher
Add MockSessionWatcher for testing

Fixes race condition between session context cancellation and internal loop exit. Eliminates goroutine leak by providing explicit lifecycle management.

sre-ci-robot · 2025-11-17T11:01:24Z

[ci-v2-notice]
Notice: We are gradually rolling out the new ci-v2 system.

Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

/ci-rerun-code-check // for ci-v2/code-check
/ci-rerun-build // for ci-v2/build
/ci-rerun-ut-integration // for ci-v2/ut-integration
/ci-rerun-ut-go // for ci-v2/ut-go
/ci-rerun-ut-cpp // for ci-v2/ut-cpp
/ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp
/ci-rerun-e2e-arm // for ci-v2/e2e-arm

If you have any questions or requests, please contact @zhikunyao.

congqixia · 2025-11-17T11:32:05Z

/ci-rerun-ut-go

mergify · 2025-11-17T11:42:41Z

@congqixia cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2025-11-17T11:49:46Z

/run-cpu-e2e

codecov · 2025-11-17T11:58:35Z

Codecov Report

❌ Patch coverage is 95.65217% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.50%. Comparing base (caed0fe) to head (f6a9728).
⚠️ Report is 31 commits behind head on master.

Files with missing lines	Patch %	Lines
cmd/tools/migration/migration/runner.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #45627       +/-   ##
===========================================
- Coverage   83.18%   76.50%    -6.68%     
===========================================
  Files         521     1875     +1354     
  Lines       81313   292178   +210865     
===========================================
+ Hits        67642   223539   +155897     
- Misses      13671    61240    +47569     
- Partials        0     7399     +7399

Components	Coverage Δ
Client	`78.17% <ø> (∅)`
Core	`83.19% <98.38%> (+0.01%)`	⬆️
Go	`74.62% <95.52%> (∅)`

Files with missing lines	Coverage Δ
internal/datacoord/server.go	`68.00% <100.00%> (ø)`
internal/distributed/connection_manager.go	`71.27% <100.00%> (ø)`
internal/querycoordv2/server.go	`76.03% <100.00%> (ø)`
internal/util/sessionutil/session_util.go	`75.59% <100.00%> (ø)`
cmd/tools/migration/migration/runner.go	`0.00% <0.00%> (ø)`

... and 1349 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mergify · 2025-11-17T12:54:12Z

@congqixia cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-11-17T18:16:28Z

@congqixia cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-11-18T04:42:36Z

@congqixia cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

…ordv2 Related to milvus-io#44620 Related to unstable ut "internal/querycoordv2 TestServer/TestNodeUp" Introduce SessionWatcher interface to fix race condition and goroutine leak that caused unstable unit test TestServer/TestNodeUp. Changes: - Add SessionWatcher interface with EventChannel() and Stop() methods - Refactor WatchServices() to return SessionWatcher instead of raw channel - Fix cleanup order in QueryCoordV2: stop watcher before session - Update DataCoord, ConnectionManager to use SessionWatcher - Add MockSessionWatcher for testing Fixes race condition between session context cancellation and internal loop exit. Eliminates goroutine leak by providing explicit lifecycle management. Signed-off-by: Congqi Xia <[email protected]>

Signed-off-by: Congqi Xia <[email protected]>

sre-ci-robot · 2025-11-20T05:07:33Z

[ci-v2-notice]
Notice: We are gradually rolling out the new ci-v2 system.

Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

/ci-rerun-code-check // for ci-v2/code-check
/ci-rerun-build // for ci-v2/build
/ci-rerun-ut-integration // for ci-v2/ut-integration
/ci-rerun-ut-go // for ci-v2/ut-go
/ci-rerun-ut-cpp // for ci-v2/ut-cpp
/ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp
/ci-rerun-e2e-arm // for ci-v2/e2e-arm

If you have any questions or requests, please contact @zhikunyao.

internal/datacoord/server.go

Signed-off-by: Congqi Xia <[email protected]>

congqixia · 2025-11-21T03:58:22Z

/ci-rerun-ut-integration

mergify · 2025-11-21T04:19:29Z

@congqixia cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2025-11-21T06:07:26Z

/run-cpu-e2e

congqixia · 2025-11-21T06:11:15Z

/ci-rerun-ut-integration

congqixia · 2025-11-21T07:50:37Z

/ci-rerun-ut-integration

liliu-z

/lgtm
/approve

sre-ci-robot · 2025-11-21T10:31:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, liliu-z

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [congqixia,liliu-z]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot added area/compilation approved labels Nov 17, 2025

sre-ci-robot requested review from bigsheeper and cydrain November 17, 2025 11:00

sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Nov 17, 2025

mergify bot added dco-passed DCO check passed. kind/bug Issues or changes related a bug labels Nov 17, 2025

congqixia force-pushed the fix/refine_sessionwatcher_unstable_ut branch from c06be7a to d95492a Compare November 18, 2025 02:57

congqixia mentioned this pull request Nov 18, 2025

[Enhancement]: ci-v2 unstable case umbrella issue #44620

Open

1 task

congqixia added 2 commits November 18, 2025 15:50

Add mutex prevent data race

9062071

Signed-off-by: Congqi Xia <[email protected]>

congqixia force-pushed the fix/refine_sessionwatcher_unstable_ut branch from d95492a to 9062071 Compare November 18, 2025 07:50

liliu-z reviewed Nov 20, 2025

View reviewed changes

internal/datacoord/server.go Outdated Show resolved Hide resolved

Resolve comment

f6a9728

Signed-off-by: Congqi Xia <[email protected]>

mergify bot added the ci-passed label Nov 21, 2025

liliu-z reviewed Nov 21, 2025

View reviewed changes

sre-ci-robot assigned liliu-z Nov 21, 2025

sre-ci-robot added the lgtm label Nov 21, 2025

sre-ci-robot merged commit f51fcc0 into milvus-io:master Nov 21, 2025
17 of 20 checks passed

fix: resolve SessionWatcher goroutine leak and unstable UT in querycoordv2 #45627

fix: resolve SessionWatcher goroutine leak and unstable UT in querycoordv2 #45627

Uh oh!

Conversation

congqixia commented Nov 17, 2025

Uh oh!

sre-ci-robot commented Nov 17, 2025

Uh oh!

congqixia commented Nov 17, 2025

Uh oh!

mergify bot commented Nov 17, 2025

Uh oh!

congqixia commented Nov 17, 2025

Uh oh!

codecov bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mergify bot commented Nov 17, 2025

Uh oh!

mergify bot commented Nov 17, 2025

Uh oh!

mergify bot commented Nov 18, 2025

Uh oh!

sre-ci-robot commented Nov 20, 2025

Uh oh!

Uh oh!

congqixia commented Nov 21, 2025

Uh oh!

mergify bot commented Nov 21, 2025

Uh oh!

congqixia commented Nov 21, 2025

Uh oh!

congqixia commented Nov 21, 2025

Uh oh!

congqixia commented Nov 21, 2025

Uh oh!

liliu-z left a comment

Choose a reason for hiding this comment

Uh oh!

sre-ci-robot commented Nov 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Nov 17, 2025 •

edited

Loading