Add Grok Bot remote MCP harness (streamable HTTP pilot)

[professorpalmer]

Summary

• Add authenticated remote MCP transport (puppetmaster/mcp_remote.py) wrapping the existing tool handlers over Streamable HTTP (/mcp) plus legacy SSE (/sse + /message), so Grok Bot can act as a pilot against Puppetmaster’s durable job/artifact runtime. Stdio MCP is unchanged.
• Default --scope supervise exposes doctor / start review|plan|swarm / status|logs|live artifacts|show / CodeGraph reads; implement/edit/browser/mutating tools stay off until --scope implement. Bearer auth is mandatory; Origin guard, per-IP rate limit, and JSONL audit ship with the server.
• CLI: python -m puppetmaster mcp serve-remote (+ puppetmaster-mcp-remote). Docs: docs/GROK_BOT.md. Optional PoC: ./scripts/grok-bot-remote-poc.sh / make grok-bot-poc. Version 1.22.0.
• Bulletproof Grok Bot handshake (live-proven: Plugins connected / tools=50) tip b00adfa:
  1. Echo any non-empty client protocolVersion (default only if missing — no allowlist).
  2. Minimal capabilities {"tools":{"listChanged":false},"logging":{}} — no experimental, no instructions.
  3. build_remote_initialize_result never calls stdio handle_message.
  4. Dual Accept → application/json; SSE only when text/event-stream without JSON.
  5. GET /mcp long-lived SSE keepalives (no instant Connection: close).
  6. remote_tool_to_json for remote tools/list: desc ≤280, prop desc ≤120, simple schema fields, additionalProperties: false, required ⊆ properties.
  7. CORS expose Mcp-Session-Id.

This is not a grok-bot worker adapter. Stdio tool surface and worker adapters are untouched.

Test plan

• tests/test_mcp_remote.py — auth, scope, dual-Accept → JSON, SSE-only → SSE, protocol echo (incl. unknown version), minimal caps, remote_tool_to_json compaction / additionalProperties: false
• tests/test_mcp_remote_e2e.py — full HTTP loop + GrokBotHandshakeRegressionTests (initialize echo + dual Accept JSON + GET keepalive + compact tools/list)
• tests/test_harness_regression_guards.py — stdio tool floor + adapters inventory (no grok-bot)
• Diff containment: no adapter/sqlite/router/mcp_server.py handler edits
• Live Grok Bot AddMcpServer against tunnel: Plugins connected, tools=50 (was Failed to load / tools=0)

Follow-ups

• Hosted HTTPS deploy / marketplace plugin packaging for Grok Bot
• OAuth / multi-tenant auth if a shared hosted endpoint becomes real
• A leased grok-bot worker adapter only if/when Grok Bot publishes dispatch

  

[cursor]

Follow-up validation (no feature expansion)

Regression / containment proof on cursor/grok-bot-remote-mcp-647f. No Grok Bot tunnel e2e (no live connector URL in this agent).

1) Diff surface (non-doc code only)

File	Role
puppetmaster/mcp_remote.py	new remote transport
puppetmaster/cli/_parser.py	additive mcp serve-remote flags only
puppetmaster/cli/commands_mcp.py	additive _run_mcp_serve_remote dispatch only
puppetmaster/__init__.py	version 1.21.14 → 1.22.0
pyproject.toml	version + puppetmaster-mcp-remote script
tests/test_mcp_remote.py	new tests

Untouched (confirmed via git diff --name-only origin/main...HEAD):

• no puppetmaster/mcp_server.py
• no puppetmaster/adapters/**
• no sqlite_store.py / schema
• no router.py / model_registry.py / workers.py

Remote imports stdio handlers (handle_message, tools, tool_to_json, …) but does not edit their bodies.

2) Full test suite (CI command)

python -m unittest discover -s tests -v
# Ran 1784 tests in 119.636s
# FAILED (failures=1, skipped=2)

• Failure: test_agentic_standalone.AgenticLoopTests.test_implement_verify_bounces_failure_then_accepts_on_pass (degraded ≠ passed)
• Same failure reproduces on origin/main — pre-existing, outside this PR’s diff surface
• GitHub CI for this PR: 8/8 green (ubuntu 3.9/3.12, macOS 3.12, Windows 3.12)

3) Stdio MCP still works (Cursor path)

Subprocess python -m puppetmaster.mcp_server JSON-RPC:

• initialize → serverInfo.name=puppetmaster, version=1.22.0
• tools/list → 50 tools
• puppetmaster_doctor: present
• puppetmaster_start_implement: present
• puppetmaster_edit: present

Installer handshake (handshake_mcp_server): ok=True, tool_count=50.

Contrast remote supervise: 32 tools, start_implement absent (scope gate only on remote).

4) Adapters inventory

python -m puppetmaster adapters →
local, agentic, shell, cursor, claude-code, openai, codex, hermes
No grok-bot adapter.

5) CLI non-regression

• puppetmaster --help → exit 0 (full subcommand set intact)
• puppetmaster mcp --help → {list,doctor,cleanup,serve-remote} — serve-remote additive only

PASS/FAIL matrix

Check	Result
CI (PR matrix)	PASS (8/8)
Full local unittest	PASS* (1784 ran; 1 pre-existing fail on main; 2 skipped)
Stdio MCP smoke	PASS (50 tools; implement present)
Adapters inventory	PASS (no grok-bot)
Diff containment	PASS (stdio/adapters/sqlite/router untouched)

*Local full-suite asterisk: one agentic test fails identically on main; CI still green. Not introduced by remote MCP.

[cursor]

Pushed: e2e + regression guards (d86509a)

Expanded the same branch — no feature-surface change beyond tests/docs/script.

Added

• tests/test_mcp_remote_e2e.py — ephemeral serve-remote, real HTTP MCP loop: initialize → tools/list → doctor (real run_cli) → start_cursor_swarm (platform-lock + start_cli mocked; config write real) → status/show against seeded SQLite → 401 + remote_scope_denied for implement. No cloudflared in CI.
• tests/test_harness_regression_guards.py — stdio tool floor + required names (doctor + start_implement); adapter inventory must include cursor/claude-code/openai/codex/hermes/agentic/shell and must not include grok-bot; remote supervise ⊂ stdio.
• docs/GROK_BOT.md — exact AddMcpServer steps (URL …/mcp, Bearer header, supervise default).
• scripts/grok-bot-remote-poc.sh + make grok-bot-poc — optional local tunnel PoC helper.

Local proof (this push)

python -m unittest tests.test_mcp_remote_e2e tests.test_harness_regression_guards tests.test_mcp_remote -v
Ran 31 tests in ~5.4s — OK

Awaiting CI re-run on this commit.

[cursor]

Hotfix pushed: Grok Bot handshake (33073d4)

Live symptom matched: initialize 200s through tunnel, tools=0, no tools/list in audit.

Root causes fixed

1. protocolVersion echo — remote initialize no longer hardcodes 2024-11-05; echoes client request (2025-03-26 / 2025-06-18 / 2024-11-05).
2. Prefer SSE when Accept lists text/event-stream (Cursor/SDK default); curl with JSON-only Accept still gets JSON.
3. CORS Access-Control-Expose-Headers: Mcp-Session-Id so browser clients can read the session header (without this: re-initialize loop).
4. capabilities.tools: {listChanged: false} — non-empty tools capability object.

How to point Grok Bot (retry)

export PUPPETMASTER_MCP_TOKEN='…'
python -m puppetmaster mcp serve-remote \
  --host 127.0.0.1 --port 8743 \
  --scope implement --allow-origin '*'
# other terminal:
cloudflared tunnel --url http://127.0.0.1:8743

AddMcpServer:

• URL: https://<tunnel-host>/mcp ← must be /mcp, not /sse or bare host
• Header: Authorization: Bearer <token>

Local proof

python -m unittest tests.test_mcp_remote tests.test_mcp_remote_e2e tests.test_harness_regression_guards -q
Ran 34 tests — OK

Includes GrokBotHandshakeRegressionTests.test_cursor_grok_streamable_handshake_reaches_tools_list.

[cursor]

Initialize body harden pushed

build_remote_initialize_result() now owns remote initialize — it does not call stdio handle_message (which hardcodes 2024-11-05 + tools: {}).

Result body contract

• result.protocolVersion = client params.protocolVersion (fallback 2025-03-26)
• result.capabilities.tools = {"listChanged": false} (non-empty)
• CORS expose Mcp-Session-Id + SSE-prefer unchanged
• Audit detail on initialize: protocolVersion=2025-03-26;tools_cap=...

Tests

GrokBotHandshakeRegressionTests asserts the JSON-RPC result body (not only headers): echoed 2025-03-26, rejects 2024-11-05, non-empty tools cap. Also compares against stdio handle_message defaults to prove divergence.

python -m unittest tests.test_mcp_remote tests.test_mcp_remote_e2e -q  → OK

Operator note

If live smoke still shows 2024-11-05, the old serve-remote process is still running — kill and restart after pull:

# restart
python -m puppetmaster mcp serve-remote --scope implement --allow-origin '*'
# curl check — must print 2025-03-26
curl -sS -X POST "$URL/mcp" \
  -H "Authorization: Bearer $PUPPETMASTER_MCP_TOKEN" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' \
  | python -c 'import sys,json; print(json.load(sys.stdin)["result"]["protocolVersion"], json.load(open("/dev/stdin")) if False else "")'

Simpler check: look for audit line protocolVersion=2025-03-26;tools_cap=... on initialize.

[cursor]

Dual-Accept → JSON fix (2f524f7)

POST /mcp Accept rule: JSON if Accept includes application/json, is empty, or */*; SSE only if Accept lists text/event-stream without application/json. Dual application/json, text/event-stream (Grok Bot) now gets Content-Type: application/json. Protocol echo + tools cap + CORS expose unchanged. Tests updated (dual→JSON, SSE-only→SSE, JSON-only→JSON). Restart serve-remote after pull.

[cursor]

GET /mcp long-lived stream + Accept JSON preference

Accept (already on branch, reconfirmed): JSON if Accept has application/json / empty / */*; SSE only if text/event-stream without JSON.

GET /mcp: no longer writes one comment and Connection: close. Holds open with SSE comment keepalives (default 15s, max 300s), CORS expose intact, audits mcp_get_stream_open / mcp_get_stream_close. Unknown Mcp-Session-Id → 404; missing session still allowed.

Tests: dual Accept → JSON; SSE-only → SSE; GET stream receives stream open + keepalive; initialize body still echoes protocol + tools.listChanged.

Restart serve-remote after pull. If Plugins still flaps with a huge implement tool list, try --scope supervise first (docs note only — tools not stripped).