Skip to content

fix(mcp): show startup progress as each server connects - #5897

Merged
Hmbown merged 3 commits into
mainfrom
fix/mcp-live-boot-progress-0913-20260905
Sep 6, 2026
Merged

fix(mcp): show startup progress as each server connects#5897
Hmbown merged 3 commits into
mainfrom
fix/mcp-live-boot-progress-0913-20260905

Conversation

@Hmbown

@Hmbown Hmbown commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Startup buffered every MCP connection result until the slowest batch finished, so the TUI could keep showing “20 connecting” even after individual servers were ready. Ready tools were also unavailable in the pool during that wait.

Consume each owned connection task as it completes, update the engine-owned pool under a short lock, and emit per-server progress immediately. Both startup and explicit connect-all use the same bounded task set. Preserve the eight-connection limit, catalog generations, plugin-authority validation, required-server errors, and cancellation when the task set is dropped.

Fixes #5887. The deterministic fixture proves this startup batching defect; it does not reproduce every server in the founder’s private configuration.

Validation: 157/157 focused MCP and engine nextest tests passed with the documented 16 MiB test stack; cargo check, Clippy with -D warnings, and formatting pass. A real engine event-loop fixture holds one stdio server until a release file is written and verifies that the other server is already visible and the connecting count has dropped before that release. Boot completion, retry ownership, stale-generation handling, catalog and authority tests pass.

An initial broad libtest substring filter also selected an unrelated setup test and aborted on its default-stack overflow; the focused run uses the repository’s documented stack setting. Full workspace tests were not run locally.

Based on the bundle relocation in #5890. This PR is v0.9.13 work and does not retag or republish v0.9.12.


Note

Medium Risk
Changes MCP connection lifecycle, config reload races during concurrent boot, and LLM tool-catalog/prefix pinning timing—important for correctness but covered by focused integration tests.

Overview
MCP session boot no longer waits for the slowest handshake before updating the UI or tool catalog. The engine now consumes spawn_pending_connects tasks as each finishes, stores ready connections under a short pool lock, and emits McpSessionBoot progress with an updated connecting list after every completion instead of buffering until the batch ends.

Ready tools are exposed while optional servers are still connecting, and pending_prefix_change_reason is set to mcp-session-boot on progress, finished boot, and early mcp_tools() snapshots so the KV prefix refresh is scheduled even when a connection becomes ready before its mailbox event.

store_ready_connection now returns Result and rejects connections whose catalog generation no longer matches the pool (with plugin validation on store). Mid-boot config reloads call reload_if_config_changed, abort remaining connect tasks, and avoid letting stale handshakes overwrite the new configuration. Connect tasks are wrapped with panic catching via FutureExt.

New engine and pool tests cover incremental boot visibility, config invalidation during handshake, and prefix-change ordering relative to progress events.

Reviewed by Cursor Bugbot for commit d716c0c. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI lite review requested due to automatic review settings September 5, 2026 12:04
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_d0f84519-0fd0-49de-8b93-d0eea008f5e7)

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-05T12:11:40.070113Z 4530a19 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Hmbown
Hmbown changed the base branch from fix/v0912-crate-bundle-20260905 to main September 5, 2026 12:07
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_e818c6b8-8f33-4b8e-91f9-f57adb9ba9ae)

@claude

claude Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Claude finished @Hmbown's task in 2m 35s —— View job


Review of #5897 — show MCP startup progress as each server connects

  • Gather context (diff, changed files, PR description)
  • Review crates/tui/src/core/engine.rs streaming connect logic
  • Review crates/tui/src/core/engine/tests.rs new regression test
  • Review crates/tui/src/mcp.rs (spawn_pending_connects, connect_all)
  • Review plugin relocation (builtin.rs, manifest.rs, extensions.rs, .gitignore, ci.yml)
  • Check provider/model-route safety implications
  • Post final review

Summary

The core change is sound. McpPool::spawn_pending_connects (crates/tui/src/mcp.rs:3096) now returns the JoinSet instead of awaiting it internally, and both callers (start_mcp_session_boot in crates/tui/src/core/engine.rs:6483-6524, and McpPool::connect_all in crates/tui/src/mcp.rs:3183-3196) drain it with join_next(), taking the pool lock only per-completion instead of once for the whole batch. That correctly fixes the reported bug: a fast server's ready state and tools are now visible (and progress emitted) immediately, rather than withheld until the slowest server in the batch finishes. The eight-connection semaphore (CONNECT_CONCURRENCY = 8, mcp.rs:3013), the required-server sweep, and the two-pass config-reload reconciliation in connect_all are all preserved.

The new test mcp_boot_reports_ready_server_before_stalled_server_finishes (crates/tui/src/core/engine/tests.rs) is a genuine regression test: it uses a real stdio server fixture where the slow server blocks on initialize until a release file appears, and asserts the fast server is already connected and slow is still connecting before the release file is written. This would fail under the old batch-buffered behavior and passes under the fix — good evidence, not just a pinned-implementation test.

Plugin relocation (plugins/computer-usecrates/tui/plugins/computer-use, referenced as based on #5890) is consistent: verified no stale root-level plugins/computer-use references remain anywhere in the tree (builtin.rs's include_str! path math, manifest.rs's test fixture path, extensions.rs's source_reference, .gitignore, and ci.yml's npm test cwd were all updated correctly to match the new location).

No provider/model-prefix routing code is touched by this PR — the EPIC #2608 invariant doesn't apply here.

Minor / non-blocking notes

  1. Small duplication — the JoinError-normalization one-liner

    .unwrap_or_else(|error| ("connection task".to_string(), Err(error.into())));

    now appears twice (engine.rs:6496 and mcp.rs:3192) since the join loop was pushed into both callers. Previously it lived once inside connect_pending_concurrently. Could be a tiny shared helper (e.g. on JoinSet) but it's one line at two call sites, so not worth blocking on.

  2. Pre-existing, out of scopespawn_pending_connects still tags connection.catalog_generation = catalog_generation inside the connect future (mcp.rs:3115), using the generation captured at spawn time. Both call sites immediately overwrite that field via store_ready_connection, which re-tags it with the pool's current generation at insertion time (mcp.rs:2937). So the spawn-time assignment is effectively dead — this predates this PR and isn't part of the diff, just flagging for a future cleanup pass since it's easy to misread as meaningful staleness handling.

Nothing else stood out as a correctness, race-condition, or safety issue. I was not able to run cargo test/clippy in this sandbox (tool approval required for cargo invocations wasn't available), so this review is based on manual reading of the diff and cross-referencing call sites; the PR description's own reported validation (157/157 focused nextest, clippy -D warnings, fmt) is the build/test evidence for this change.

@Hmbown Hmbown added this to the v0.9.13 milestone Sep 5, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes concurrency/timing in MCP boot and tool-pool visibility, which is correctness-sensitive and best validated with final human review despite good targeted regression coverage.

Pull request overview

Improves MCP session boot responsiveness by applying connection results as each server finishes handshaking (instead of buffering until the slowest finishes), so the TUI’s “connecting” count and available tool pool reflect real-time progress during startup.

Changes:

  • Replaced the “connect and await all” helper with spawn_pending_connects, returning a bounded-concurrency JoinSet that callers can join_next() incrementally.
  • Updated MCP session boot (and pool connect flow) to store each successful connection under a short lock and emit per-server progress updates immediately as each task completes.
  • Added an engine integration test that stalls one stdio server during initialize and asserts the fast server becomes visible (and connecting count drops) before the slow server is released.
File summaries
File Description
crates/tui/src/mcp.rs Introduces spawn_pending_connects (returns JoinSet) and adjusts connect flow to consume results incrementally while preserving concurrency limits and error behavior.
crates/tui/src/core/engine.rs Updates MCP session boot to join_next() each connection task, update the shared pool under a short lock, and emit per-completion progress events.
crates/tui/src/core/engine/tests.rs Adds a regression test ensuring mid-boot snapshots show early-ready servers even if another server stalls.
Review details
  • Files reviewed: 8/32 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4530a19ceb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +20790 to +20791
"fast": { "command": "node", "args": [server, "fast", release] },
"slow": { "command": "node", "args": [server, "slow", release] }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip this fixture when Node is unavailable

On Rust-only or minimal build hosts where node is not on PATH, both fixture connections fail to spawn, so the progress loop consumes the finished event, waits for its 10-second timeout, and the subsequent finished loop waits another 10 seconds before the test fails. Node is not a required dependency for Codewhale, and the existing Node-backed test in crates/tui/src/mcp/tests.rs explicitly checks availability and skips when absent; add the same guard here or use a self-contained test server so the TUI library suite remains portable.

Useful? React with 👍 / 👎.

Comment thread crates/tui/src/core/engine.rs Outdated
{
let mut pool = pool_for_task.lock().await;
match result {
Ok(connection) => pool.store_ready_connection(name, connection),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Revalidate each completed connection before storing it

If the MCP config is edited while a slow startup connection is pending, an already-ready server exposed by this change can be invoked and cause get_or_connect to reload the pool and drop connections under the old config; when the slow task later completes, this unconditional store re-inserts its stale connection and store_ready_connection stamps it with the new catalog generation. Because all_tools projects stored connections and a subsequent get_or_connect returns a ready connection without checking that its server is still configured or enabled, a removed, disabled, or authority-changed server can remain advertised and executable. Reject completions whose captured generation/config authority is stale, or reconcile the config before inserting each result.

AGENTS.md reference: AGENTS.md:L28-L29

Useful? React with 👍 / 👎.

Comment on lines +6509 to +6510
let _ = progress_tx.send(McpBootUpdate::Progress {
generation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Declare partial MCP catalog changes before the next turn

When one server completes between user turns while at least one slower server is still connecting, this progress path makes the newly ready server's tools appear in the next turn but does not set pending_prefix_change_reason; that reason is set only by the later Finished update. The next request therefore changes the session-pinned tools array as undeclared drift, which trips the C5 debug_assert! in debug builds and records an avoidable KV-cache miss in release builds. Either keep the MCP catalog frozen until boot finishes or stamp a declared prefix change whenever a progress update exposes new tools.

AGENTS.md reference: AGENTS.md:L136-L139

Useful? React with 👍 / 👎.

@codewhale-agent codewhale-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codewhale review

PR replaces bulk MCP connect collection with per-completion JoinSet processing so the engine emits progress and stores ready connections as each server finishes, and relocates the computer-use plugin bundle to crates/tui/plugins. The change is generally well-formed; the main concerns are panic bookkeeping and test coverage for tool availability.

Findings

  • [WARNING] Panicked connect tasks leave the real server in the connecting list (crates/tui/src/core/engine.rs:6497)
    When a spawned connect task panics, join_next returns a JoinError and the code synthesizes the name "connection task". remaining.retain(|pending_name| pending_name != &name) does not remove the actual pending server name, so subsequent McpBootUpdate::Progress events can continue to report that server as connecting and connection_errors is keyed under the synthetic name. This was present before, but the new per-completion progress makes the stale UI state directly visible.

Suggestions

  • crates/tui/src/core/engine/tests.rs — Extend the new mcp_boot_reports_ready_server_before_stalled_server_finishes test to also assert that the fast server's tools are already available in the MCP pool before the slow server is released. The current snapshot assertion proves the progress list is fixed, but not the claimed tool-availability behavior.

Assessment

The core change correctly removes the slowest-server bottleneck and keeps the eight-connection limit intact. I would like the panicked-task bookkeeping fixed and the regression test extended to verify ready tools are admitted before slow servers finish; otherwise the implementation matches the stated goals.


Advisory review by Codewhale (codewhale review --pr 5897 --post, head d4e6c868188d67c31ddef090cee7889783a3ff1d). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.

while let Some(joined) = connects.join_next().await {
let (name, result) = joined
.unwrap_or_else(|error| ("connection task".to_string(), Err(error.into())));
remaining.retain(|pending_name| pending_name != &name);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] Panicked connect tasks leave the real server in the connecting list

When a spawned connect task panics, join_next returns a JoinError and the code synthesizes the name "connection task". remaining.retain(|pending_name| pending_name != &name) does not remove the actual pending server name, so subsequent McpBootUpdate::Progress events can continue to report that server as connecting and connection_errors is keyed under the synthetic name. This was present before, but the new per-completion progress makes the stale UI state directly visible.

@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_d3c1fa23-39a5-490f-989b-2dbfdd34d291)

@claude

claude Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Claude finished @Hmbown's task in 3m 33s —— View job


Review of #5897 — show MCP startup progress as each server connects

  • Gather context (diff, PR description, prior automated review comments)
  • Review crates/tui/src/core/engine.rs streaming connect/progress logic
  • Review crates/tui/src/mcp.rs (spawn_pending_connects, connect_all)
  • Review crates/tui/src/core/engine/tests.rs new regression tests
  • Review plugin relocation diffs (unchanged since prior review — builtin.rs, manifest.rs, extensions.rs, ci.yml, .gitignore)
  • Check provider/model-route safety implications (EPIC v0.8.65 EPIC: Separate provider facts, model facts, offerings, and route resolution #2608 invariant)
  • Cross-check prior automated findings (Codex, Copilot, codewhale-agent) for validity against latest commit
  • Post final review

Summary

Note: origin/main and this branch's original merge-base (82b2efa) have no common ancestor in the local clone (history on main appears to have been rewritten since this PR's merge commit), so I diffed against 82b2efaef2de52173a2281e9f58048a432d41146...HEAD, which reproduces the PR's reported +304/-77 across 33 files exactly — that's the correct diff for this review.

The newest commit, 9d05c12 "reject stale boot connections and declare live catalog changes," directly answers every substantive finding raised by the earlier automated reviews on this PR:

  1. Codex P1 (revalidate before storing / stale-config re-insertion)store_ready_connection (crates/tui/src/mcp.rs:2938) now returns Result<()> and rejects a connection whose captured catalog_generation no longer matches the pool's current one, and re-checks plugin authority via validate_before_use before inserting. The boot loop (engine.rs:6497-6522) also proactively calls pool.reload_if_config_changed() before storing each result and, if the generation moved, aborts the remaining JoinSet and clears errors rather than admitting stale connections. stale_handshake_cannot_be_restamped_after_config_reload (mcp/tests.rs) is a direct unit test for this, including confirming the rejected connection is dropped (no leak).
  2. Codex P1 (declare partial catalog changes)pending_prefix_change_reason is now set both wherever a Progress/Finished update lands (engine.rs:6332, 6386) and in mcp_tools() when returning a snapshot mid-boot (engine.rs:6637), with a new test (mcp_boot_catalog_refresh_declares_prefix_before_mailbox_delivery) covering both paths.
  3. Codex P2 (Node dependency in the new fixture test) — the new tests now check node --version and skip with an eprintln! if unavailable, matching the existing pattern in mcp/tests.rs.
  4. codewhale-agent WARNING (panicked task loses the real server name)spawn_pending_connects now wraps the connect future itself in catch_unwind (mcp.rs:3122-3138) instead of relying on JoinSet's JoinError, so a panic inside the handshake surfaces as Err under the real server name rather than the synthetic "connection task" placeholder. The placeholder path only remains for genuine JoinError (external abort/cancellation), which is expected and used deliberately by the new stale-generation abort_all() path.
  5. codewhale-agent suggestion (assert tool availability, not just progress list) — the regression test now asserts ready_tools contains the fast server's tool and not the slow server's before release, plus a new sibling test (mcp_boot_does_not_restore_servers_removed_during_handshake) verifying a server disabled mid-handshake is neither connected nor exposed in tools.

I re-read these fixes against their call sites rather than trusting the commit message; the generation/authority checks line up correctly (no intervening .await between setting connection.catalog_generation and the store call in get_or_connect/retry_connection, so those checks are trivially satisfied there, and the meaningful check happens in the async boot/connect-all loops where the race is real).

Minor / non-blocking

  • get_or_connect and retry_connection (mcp.rs:2868, 2932) don't call note_connect_failure when the new store_ready_connection rejection fires (e.g., plugin authority revoked between handshake start and completion). That skips the backoff/cooldown bookkeeping, so a caller could retry that server on every subsequent tool call with no backoff, unlike every other failure path in this file. Low severity (bounded by call frequency, not a crash/security issue) but worth a follow-up for consistency.
  • Small duplication remains: the JoinError-normalization line (.unwrap_or_else(|error| ("connection task".to_string(), Err(error.into())))) appears at both engine.rs:6498 and mcp.rs:3211 — pre-existing observation from the earlier review, still true, still not worth blocking on.

No correctness, race, or panic issues found beyond the above. Provider/model routing (EPIC #2608 invariant) is untouched by this PR — confirmed via git diff | grep for provider-prefix patterns, no hits. I was unable to run cargo check/clippy/tests in this sandbox (tool approval for cargo invocations was not granted), so this review is based on manual reading of the diff and cross-referencing call sites; the PR's own reported validation (160 focused nextest tests, clippy -D warnings, fmt) is the build/test evidence for this change.

@Hmbown

Hmbown commented Sep 5, 2026

Copy link
Copy Markdown
Owner Author

Review follow-up is pushed at 9d05c12. Background completion re-reads configuration, cancels obsolete batches, and refuses to re-stamp stale connection generations. Plugin authority is checked before insertion. Progress application, queue draining, and the in-flight tool snapshot all declare the catalog refresh. Handshake panics retain their server name, and Node fixtures skip cleanly when Node is unavailable.

160 focused tests pass, including ready tool availability before a stalled handshake completes, config removal/disable during that handshake, stale generation rejection, and prefix declarations. Cargo check, clippy with warnings denied, formatting, and diff checks pass. Hosted CI is running on this new head.

@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_2f6d1501-7015-41e6-8320-0278981b398d)

@claude

claude Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Claude finished @Hmbown's task in 1m 56s —— View job


Review of #5897 — show MCP startup progress as each server connects

  • Gather context (diff since last review, latest commit c92ebf7)
  • Review latest commit's changes (Node fixture tracing diagnostics)
  • Re-check prior findings (Codex P1s, codewhale-agent WARNING) are actually resolved in 9d05c12
  • Check provider/model-route safety implications
  • Post final review

Summary

Diffed 82b2efa...HEAD (the correct merge-base per the prior review, since origin/main...HEAD has no common ancestor in this clone) to reproduce the reported +304/-77 across 33 files, then focused on what changed since the last automated review: just the newest commit, c92ebf7.

c92ebf7 is a one-line, low-risk fix: it swaps eprintln!("skipping MCP stdio fixture because node is unavailable") for tracing::warn!(...) in crates/tui/src/core/engine/tests.rs:20776, satisfying clippy::print_stderr. Correct and unremarkable.

I also independently re-verified (by reading the actual diffs, not just trusting the prior comments) that 9d05c12 genuinely resolves the substantive findings raised by earlier reviews on this PR:

  1. Stale-connection rejectionstore_ready_connection (crates/tui/src/mcp.rs:2938) now returns Result and rejects a connection whose captured catalog_generation no longer matches current_catalog_generation(), and re-validates plugin authority via validate_before_use before inserting. The boot loop (engine.rs:6497-6522) calls pool.reload_if_config_changed() before each store and aborts the remaining JoinSet on a generation mismatch, clearing partial errors rather than admitting stale results. Backed by a real unit test, stale_handshake_cannot_be_restamped_after_config_reload (mcp/tests.rs), which asserts the connection is both rejected and dropped (DropCountingTransport count = 1, no leak).
  2. Undeclared partial catalog changespending_prefix_change_reason is now set at both Progress and Finished update sites (engine.rs:6332, 6386) and in mcp_tools() when it snapshots mid-boot (engine.rs:6637), with mcp_boot_catalog_refresh_declares_prefix_before_mailbox_delivery covering both paths directly.
  3. Panicked task losing the real server namespawn_pending_connects (mcp.rs:3122-3138) now wraps the connect future in catch_unwind itself, so a handshake panic surfaces as Err under the real server name. The "connection task" placeholder in the join_next() callers is now only reached for genuine JoinError (the deliberate abort_all() cancellation path on stale generation), which is the correct, expected use.
  4. Tool-availability assertion — the regression test now asserts ready_tools contains mcp_fast_ready and not mcp_slow_ready before the release file is written, and a sibling test (mcp_boot_does_not_restore_servers_removed_during_handshake) verifies a server disabled mid-handshake ends up disconnected with an empty tool pool — not just absent from a progress list.

Plugin relocation (plugins/computer-usecrates/tui/plugins/computer-use) is unchanged since the last review and still checks out: include_str! path math in builtin.rs (../../plugins/computer-use/ from crates/tui/src/plugins/), the manifest test's CARGO_MANIFEST_DIR-relative path, extensions.rs's source_reference, .gitignore, and ci.yml's npm test cwd are all consistent with the new location.

No provider/model-prefix routing code is touched (git diff | grep -iE "deepseek|anthropic/|openai/|qwen/|provider_prefix" — no hits); the EPIC #2608 invariant doesn't apply here.

Minor / still open, non-blocking

  • get_or_connect and retry_connection (mcp.rs:2868, 2932) still propagate a store_ready_connection rejection via ? without calling note_connect_failure, unlike every other failure path in this file (and unlike the boot loop / connect_all, which both call it correctly on rejection). In practice the generation check can't trip here (no .await between stamping catalog_generation and storing), but a plugin-authority revocation landing in the .await window during connect_with_policy would skip backoff bookkeeping, letting that server retry on every subsequent tool call with no cooldown. Same observation as the prior review round — still true, still low severity, worth a follow-up rather than blocking this PR.
  • The JoinError-normalization one-liner (.unwrap_or_else(|error| ("connection task".to_string(), Err(error.into())))) still appears at both engine.rs:6498 and mcp.rs:3211. Pre-existing observation, one line at two call sites, not worth blocking on.

Verdict

Nothing new to flag. This round's only change (c92ebf7) is a trivial clippy fix, and the prior round's P1 findings (stale-connection re-insertion, undeclared prefix drift, panic name loss) are genuinely fixed with direct test coverage, not just asserted in the commit message. I was unable to run cargo check/clippy/tests in this sandbox (no tool approval for cargo invocations), so this review is based on manual reading of the diff and call sites; the PR's own reported validation (160 focused nextest tests, clippy -D warnings, fmt) is the build/test evidence for this change.

@codewhale-agent codewhale-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codewhale review

PR #5897 replaces the batched MCP connection wait with a JoinSet consumed as tasks complete, so progress events and pool tools are emitted as each server finishes. It also tightens stale-generation handling in store_ready_connection and updates computer-use bundle paths. The core approach is sound and the added tests cover fast-before-slow visibility and stale pool insertion, but two concurrency/bookkeeping risks remain.

Findings

  • [WARNING] Pool mutex is held across an await in the incremental boot loop (crates/tui/src/core/engine.rs:6506)
    The engine boot task locks pool_for_task and then awaits pool.reload_if_config_changed() while still holding the guard. This can block mcp_tools() snapshots, TUI progress reads, and other pool users for filesystem/latency, which contradicts the short-lock intent and can stall UI during config reload.
  • [WARNING] Explicit connect-all records stale config errors as connection failures (crates/tui/src/mcp.rs:3214)
    In McpPool::connect_all_servers, a result rejected by store_ready_connection because the catalog generation changed is passed to note_connect_failure and pushed into the returned errors. That pollutes per-server backoff/auth state for servers that may be valid under the newly reloaded config; stale successes should be ignored or remaining connects aborted, as the engine boot loop does.

Assessment

The implementation is close and the test coverage is good, but the async lock hold should be scoped down and the connect-all stale-result path should avoid recording false failures before merge.


Advisory review by Codewhale (codewhale review --pr 5897 --post, head c92ebf795953318162b77d68d155f132957bbde2). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.

// A turn may have reloaded the pool while these handshakes
// were in flight. Never let their old authority or failures
// overwrite the newly installed configuration.
let reload = pool.reload_if_config_changed().await;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] Pool mutex is held across an await in the incremental boot loop

The engine boot task locks pool_for_task and then awaits pool.reload_if_config_changed() while still holding the guard. This can block mcp_tools() snapshots, TUI progress reads, and other pool users for filesystem/latency, which contradicts the short-lock intent and can stall UI during config reload.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the callee: reload_if_config_changed is an async wrapper whose body only calls the synchronous reload_from_config_sources(false); it contains no await and performs no handshake/network request. Its future does not park while holding the guard. The mtime/config check is intentional at insertion so a stale handshake cannot restore revoked configuration. The existing reload implementation uses synchronous filesystem reads under the same pool authority; moving those reads off-thread would be a separate reload refactor, not a correction to the claimed suspended-await behavior. The delayed-server test verifies ready tools remain available while another handshake is stalled.

Comment thread crates/tui/src/mcp.rs
let result = result
.and_then(|connection| self.store_ready_connection(name.clone(), connection));
if let Err(error) = result {
self.note_connect_failure(&name, &error);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] Explicit connect-all records stale config errors as connection failures

In McpPool::connect_all_servers, a result rejected by store_ready_connection because the catalog generation changed is passed to note_connect_failure and pushed into the returned errors. That pollutes per-server backoff/auth state for servers that may be valid under the newly reloaded config; stale successes should be ignored or remaining connects aborted, as the engine boot loop does.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In connect_all, &mut self is held for the complete batch. The catalog-generation atomic belongs to this pool (it is not a cloned Arc), and its mutation methods require borrowing this same pool; another safe caller cannot mutate it concurrently with that exclusive borrow. The config reload occurs after the batch, and that existing path clears the old pass's errors before retrying. The background engine pass is the path that releases the pool between handshakes, and it now explicitly rejects/cancels obsolete batches before recording success or failure. I do not see a reachable stale-generation/backoff case in the exclusive connect_all path.

@Hmbown
Hmbown force-pushed the fix/mcp-live-boot-progress-0913-20260905 branch from c92ebf7 to 25efa6f Compare September 6, 2026 00:07
@cursor

cursor Bot commented Sep 6, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_2ffd3c0e-acfc-4d0c-abfb-8e04cfbd4094)

@claude

claude Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Claude finished @Hmbown's task in 2m 10s —— View job


Review of #5897 — show MCP startup progress as each server connects

  • Gather context (diff since last review, confirm head vs prior automated passes)
  • Independently re-verify the P1 fixes (stale-generation rejection, prefix declaration, panic naming) against current source
  • Re-check the two open codewhale-agent WARNINGs (lock-across-await, connect_all stale-error recording) against actual code paths
  • Post final review

Summary

git diff origin/main...HEAD reproduces the PR's reported +295/-65 across exactly these 4 files, so this is the same head (25efa6f locally / c92ebf7 on GitHub) already covered by the last automated pass — no new commits to review. I re-read the relevant code directly rather than re-trusting prior summaries; findings below are from that independent pass.

Confirmed still correct:

  • store_ready_connection (crates/tui/src/mcp.rs:2938) returns Result and rejects a completion whose catalog_generation no longer matches current_catalog_generation(), re-validating plugin authority via validate_before_use before insertion. The boot loop (crates/tui/src/core/engine.rs:6497-6528) calls reload_if_config_changed() before each store and abort_all()s the remaining JoinSet on a generation mismatch, clearing connection_errors from the stale pass. stale_handshake_cannot_be_restamped_after_config_reload exercises this and also asserts the dropped connection's transport is actually closed (drops == 1).
  • pending_prefix_change_reason is now stamped at both Progress (engine.rs:6332) and Finished (engine.rs:6386) update sites, and in mcp_tools() (engine.rs:6637) when it snapshots mid-boot — covered directly by mcp_boot_catalog_refresh_declares_prefix_before_mailbox_delivery.
  • spawn_pending_connects (mcp.rs:3122-3138) wraps the connect future itself in catch_unwind, so a handshake panic now surfaces under the real server name; the "connection task" synthetic name in the join_next() callers is only reachable via a genuine JoinError from the deliberate abort_all() cancellation, and that path always breaks immediately afterward, so remaining/connecting never gets falsely reported once that fires. Verified there's no other route to a JoinError here.
  • The new regression tests (mcp_boot_reports_ready_server_before_stalled_server_finishes, mcp_boot_does_not_restore_servers_removed_during_handshake) assert on ready_tools directly (mcp_fast_ready present, mcp_slow_ready absent before release), not just the progress list, and both skip cleanly via tracing::warn! when node isn't on PATH.

Re-checked the two open codewhale-agent WARNINGs and agree with the author's rebuttals:

  • "Pool mutex held across await" (engine.rs:6506): reload_if_config_changedreload_from_config_sources(false) (mcp.rs:2696-2750) contains no .await in its body — it's a sync mtime/hash check with no network or handshake I/O. Holding the guard across it isn't a suspension point that can starve other pool users the way a real awaited I/O call would.
  • "connect_all records stale config errors as failures" (mcp.rs:3202-3217): connect_all takes &mut self for the whole batch, so nothing else can bump catalog_generation concurrently with this call — the race the finding describes isn't reachable from this exclusive-borrow path (unlike the shared, Arc<Mutex<_>>-backed boot loop, which correctly guards against it).

Still open, non-blocking (unchanged from prior rounds, already acknowledged by the author):

  • get_or_connect/retry_connection (mcp.rs:2868, 2932) propagate a store_ready_connection rejection via ? without calling note_connect_failure, so that one rejection path skips backoff bookkeeping. Narrow window, no crash/security impact.
  • The JoinError-normalization one-liner is duplicated at engine.rs:6498 and mcp.rs:3211 (previously lived once in the removed connect_pending_concurrently). One line, two call sites — not worth a shared helper.

No provider/model-prefix routing code is touched (checked for deepseek/anthropic//openai//qwen/ provider-prefix patterns in the diff — no hits), so the EPIC #2608 invariant doesn't apply here.

I could not run cargo check/clippy/cargo test in this sandbox (tool approval for cargo invocations isn't available here), so this is based on manual reading of the diff and call sites. The PR's own reported validation (160 focused nextest tests, clippy -D warnings, fmt) remains the build/test evidence for this change.

Verdict

No new or unresolved correctness issues found on this head. The substantive P1s from earlier rounds (stale-connection re-insertion, undeclared prefix drift, panic name loss) are genuinely fixed with direct test coverage, and the two most recent WARNINGs don't hold up against the actual (non-yielding reload, exclusive-borrow connect_all) code paths.
· branch fix/mcp-live-boot-progress-0913-20260905

@codewhale-agent codewhale-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codewhale review

The PR replaces batch MCP connection boot with an incremental JoinSet loop so ready servers become visible and usable as soon as their handshake completes. It also hardens store_ready_connection against stale generation and invalid plugin authority, and marks the KV-cache prefix refresh earlier. The added fixtures cover the main boot-ordering regression and stale-generation behavior.

Findings

  • [WARNING] MCP pool lock is held across an awaited config reload (crates/tui/src/core/engine.rs:6505)
    In the new per-completion boot loop, pool_for_task.lock().await is acquired and then pool.reload_if_config_changed().await is awaited while still holding the guard. A config reload can perform file I/O or other work, so this may block all MCP snapshots and tool reads for the duration of the reload, working against the intended short lock scope.
  • [INFO] store_ready_connection now depends on callers stamping catalog_generation (crates/tui/src/mcp.rs:2954)
    The function no longer sets connection.catalog_generation internally and instead rejects mismatches. The visible production call sites are updated, but the invariant is now enforced only at runtime, so any future or missed caller that passes an unstamped connection will fail with a generation error rather than a compile-time error.
  • [INFO] Incremental connect_all and JoinError/panic attribution lack focused tests (crates/tui/src/mcp/tests.rs)
    The diff changes connect_all to consume joins incrementally and routes task panics through spawn_pending_connects, but no new test appears to exercise the explicit connect-all path, its generation/stale rejection behavior, or the panic-to-error JoinSet attribution added here.

Assessment

The change is broadly correct and directly addresses the startup batching defect with meaningful integration coverage. The main concern is the pool lock being held across an async reload; no blocking correctness issue was identified in the diff.


Advisory review by Codewhale (codewhale review --pr 5897 --post, head 25efa6fe87e1959ea9852f820e4a9a5a8e57101e). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.

let mut pool = pool_for_task.lock().await;
// A turn may have reloaded the pool while these handshakes
// were in flight. Never let their old authority or failures
// overwrite the newly installed configuration.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] MCP pool lock is held across an awaited config reload

In the new per-completion boot loop, pool_for_task.lock().await is acquired and then pool.reload_if_config_changed().await is awaited while still holding the guard. A config reload can perform file I/O or other work, so this may block all MCP snapshots and tool reads for the duration of the reload, working against the intended short lock scope.

Comment thread crates/tui/src/mcp.rs
// and the cooldown with it.
self.connect_backoff.remove(&name);
if self.needs_auth_servers.remove(&name) {
self.needs_auth_generation = self.needs_auth_generation.wrapping_add(1);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[INFO] store_ready_connection now depends on callers stamping catalog_generation

The function no longer sets connection.catalog_generation internally and instead rejects mismatches. The visible production call sites are updated, but the invariant is now enforced only at runtime, so any future or missed caller that passes an unstamped connection will fail with a generation error rather than a compile-time error.

CodeWhale Bot added 3 commits September 5, 2026 20:24
Addresses #5887. Return the owned concurrent connection tasks to callers and consume each completion immediately. Keep the eight-connection limit, short pool locks, catalog generation checks, required-server errors, and cancellation-on-drop.

Validation: 157/157 focused nextest tests passed with repository-standard RUST_MIN_STACK=16777216; includes a real engine event-loop fixture holding one server while the ready server becomes visible. cargo check, Clippy -D warnings, and fmt passed. An earlier broad libtest filter hit the unrelated setup test stack overflow under the default stack; no full workspace gate claimed.
Signed-off-by: CodeWhale Bot <bot@codewhale.net>
Revalidate config before each background completion, reject obsolete connection generations, and check plugin authority before insertion. Declare boot catalog changes at progress and tool snapshot boundaries; retain server names when handshakes panic.

Validation: 160 focused nextest tests passed; cargo check and clippy -D warnings passed; cargo fmt and git diff --check passed. Node fixture verifies tools are usable before the stalled handshake completes and removed/disabled servers cannot be restored. Runtime root has no npm test/check:web scripts.
Signed-off-by: CodeWhale Bot <bot@codewhale.net>
CI correctly rejected the stderr diagnostic under clippy::print_stderr. Replace it with tracing. Validation: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings passed; cargo fmt and diff checks passed. The preceding behavioral test run passed 160/160.

Signed-off-by: CodeWhale Bot <bot@codewhale.net>
@Hmbown
Hmbown force-pushed the fix/mcp-live-boot-progress-0913-20260905 branch from 25efa6f to d716c0c Compare September 6, 2026 03:25
@cursor

cursor Bot commented Sep 6, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_45ff8562-bf7b-4109-a49e-0fe7c4258c2c)

@claude

claude Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Claude finished @Hmbown's task in 1m 50s —— View job


Review of #5897 — show MCP startup progress as each server connects

  • Gather context (diff vs origin/main, confirm current head)
  • Check what changed in the newest commit(s)
  • Re-verify open findings (lock-across-await, connect_all stale errors, panic naming) against current source
  • Post review

Summary

git diff origin/main...HEAD reproduces the PR's reported +295/-65 across exactly the 4 listed files (crates/tui/src/core/engine.rs, crates/tui/src/core/engine/tests.rs, crates/tui/src/mcp.rs, crates/tui/src/mcp/tests.rs). This branch has 3 commits total, matching what's already been through several automated review rounds — there is no new commit beyond what the most recent codewhale-agent pass (head 25efa6f) already reviewed. I re-read the current source directly rather than trusting prior summaries; findings below are from that independent pass.

Confirmed correct, with direct test coverage:

  • store_ready_connection (crates/tui/src/mcp.rs:2938) rejects a connection whose catalog_generation no longer matches current_catalog_generation(), and re-validates plugin authority via validate_before_use before inserting. The boot loop (crates/tui/src/core/engine.rs:6497-6528) reloads config before each store and abort_all()s the remaining JoinSet on a generation mismatch, clearing the stale pass's errors. stale_handshake_cannot_be_restamped_after_config_reload (mcp/tests.rs) exercises this directly, including confirming the transport is actually dropped.
  • pending_prefix_change_reason is stamped at both Progress (engine.rs:6529) and Finished update sites and in mcp_tools() (engine.rs:6638) when it snapshots mid-boot — covered by mcp_boot_catalog_refresh_declares_prefix_before_mailbox_delivery.
  • spawn_pending_connects (mcp.rs:3122-3139) wraps the connect future itself in catch_unwind, so a handshake panic surfaces under the real server name rather than the synthetic "connection task" placeholder; that placeholder is only reachable via a genuine JoinError from the deliberate abort_all() cancellation, which always breaks immediately after.
  • mcp_boot_reports_ready_server_before_stalled_server_finishes and mcp_boot_does_not_restore_servers_removed_during_handshake both assert directly on ready_tools (fast server's tool present, slow server's absent before release) and skip cleanly via tracing::warn! when node isn't on PATH — this addresses the earlier Codex P2 portability concern and the codewhale-agent tool-availability suggestion.
  • connect_all (mcp.rs:3166-3217) is called everywhere behind a MutexGuard held for its entire duration (verified at engine.rs:6642-6643, the same Arc<Mutex<McpPool>> the boot loop locks per-completion), so no concurrent caller can bump catalog_generation mid-batch. The "stale config errors recorded as failures" concern from an earlier round doesn't hold up against this exclusive-borrow path — confirmed independently, not just by re-reading the author's rebuttal.
  • The engine boot loop's pool.reload_if_config_changed().await (engine.rs:6506) resolves to a synchronous mtime/hash check with no actual suspension point (reload_from_config_sources(false) performs no .await), so holding the pool guard across it isn't the blocking-I/O-under-lock hazard the finding described.

No provider/model-prefix routing code is touched by this diff (checked for deepseek/anthropic//openai//qwen/ prefix-inference patterns — no hits), so the EPIC #2608 invariant doesn't apply here.

Still open, non-blocking

  • get_or_connect and retry_connection (mcp.rs:2868, 2932) still propagate a store_ready_connection rejection via ? without calling note_connect_failure, unlike every other failure path in this file. Narrow window (no .await between stamping catalog_generation and storing in these two call sites), no crash/security impact, but worth a follow-up for consistency.
  • spawn_pending_connects's incremental JoinSet loop in connect_all (mcp.rs:3208-3217) has no test exercising its own panic-attribution or mid-batch generation-change path directly — stale_handshake_cannot_be_restamped_after_config_reload covers store_ready_connection's rejection in isolation, but not this call site's loop.
  • The JoinError-normalization one-liner (.unwrap_or_else(|error| ("connection task".to_string(), Err(error.into())))) is still duplicated at engine.rs:6498 and mcp.rs:3209. One line, two call sites — not worth blocking on.

Verdict

No new or unresolved correctness issues on this head. The substantive P1s raised earlier in this PR's review history (stale-connection re-insertion, undeclared prefix drift, panic name loss, portable Node-fixture skip) are genuinely fixed with direct test coverage rather than just asserted, and the two most recent WARNINGs (lock-across-await, connect_all stale-error recording) don't hold up against the actual code paths. I was unable to run cargo check/clippy/cargo test in this sandbox (no tool approval for cargo invocations here), so this is based on manual reading of the diff and call sites; the PR's own reported validation (160 focused nextest tests, clippy -D warnings, fmt) remains the build/test evidence for this change.
· branch fix/mcp-live-boot-progress-0913-20260905

@codewhale-agent codewhale-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codewhale review

PR changes MCP boot to consume connection tasks as they complete, storing ready connections incrementally and emitting progress. The core direction is sound; one edge case leaves the finished boot event reporting aborted connections after a config reload.

Findings

  • [WARNING] Finished boot event can report aborted servers as still connecting after config reload (crates/tui/src/core/engine.rs:6510)
    When reload_if_config_changed detects a changed catalog or reload error, the task aborts remaining connects and breaks. The local remaining Vec is not cleared, so a subsequent Finished McpSessionBoot may include those aborted server names in connecting even though they were cancelled. The new test mcp_boot_does_not_restore_servers_removed_during_handshake does not assert final connecting list, so this is not caught. Clear remaining when aborting all pending connects.
  • [INFO] Config reload during boot is only detected when a connection completes (crates/tui/src/core/engine.rs:6506)
    The reload/generation check happens inside the join_next loop, after at least one connection task has completed. If all pending handshakes are stalled, a mid-boot config reload cannot abort the remaining tasks until one handshake finishes or times out. This may delay applying the new config for the full connect timeout. Consider a separate wake/select mechanism if reloads must abort immediately.

Suggestions

  • crates/tui/src/core/engine.rs:6510 — Clear the remaining connection list when pending connects are aborted, so the Finished boot event does not report aborted servers as still connecting.

                                connects.abort_all();
                                connection_errors.clear();
                                remaining.clear();
    

Assessment

The incremental connect handling is a good improvement and includes focused regression coverage. The main correctness gap is the stale connecting list after reload/abort; clearing it and adding a test assertion would make the change robust.


Advisory review by Codewhale (codewhale review --pr 5897 --post, head d716c0c8aa0c45b44ed966f6d64e57c9ee5516bd). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.

if reload.is_err()
|| pool.current_catalog_generation() != catalog_generation
{
connects.abort_all();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[WARNING] Finished boot event can report aborted servers as still connecting after config reload

When reload_if_config_changed detects a changed catalog or reload error, the task aborts remaining connects and breaks. The local remaining Vec is not cleared, so a subsequent Finished McpSessionBoot may include those aborted server names in connecting even though they were cancelled. The new test mcp_boot_does_not_restore_servers_removed_during_handshake does not assert final connecting list, so this is not caught. Clear remaining when aborting all pending connects.

// A turn may have reloaded the pool while these handshakes
// were in flight. Never let their old authority or failures
// overwrite the newly installed configuration.
let reload = pool.reload_if_config_changed().await;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[INFO] Config reload during boot is only detected when a connection completes

The reload/generation check happens inside the join_next loop, after at least one connection task has completed. If all pending handshakes are stalled, a mid-boot config reload cannot abort the remaining tasks until one handshake finishes or times out. This may delay applying the new config for the full connect timeout. Consider a separate wake/select mechanism if reloads must abort immediately.

Comment on lines +6510 to +6511
connects.abort_all();
connection_errors.clear();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clear the remaining connection list when pending connects are aborted, so the Finished boot event does not report aborted servers as still connecting.

Suggested change
connects.abort_all();
connection_errors.clear();
connects.abort_all();
connection_errors.clear();
remaining.clear();

@Hmbown
Hmbown merged commit 27b48b7 into main Sep 6, 2026
33 checks passed
@Hmbown
Hmbown deleted the fix/mcp-live-boot-progress-0913-20260905 branch September 6, 2026 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP startup can stay on “20 connecting” for a long time

2 participants