Skip to content

fixing MCP headers usag for Llama stack 0.4.x, adding additional e2e tests for mcp servers #1080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
blublinsky wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fixing MCP headers usag for Llama stack 0.4.x, adding additional e2e tests for mcp servers #1080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
74a4a0a
adding additional e2e tests for mcp servers
blublinsky Jan 30, 2026
661c9b2
fixed code to correctly execute e2e test for MCP
blublinsky Feb 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions dev-tools/mcp-mock-server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ This mock server helps developers:
- ✅ **HTTP & HTTPS** - Runs both protocols simultaneously for comprehensive testing
- ✅ **Header Capture** - Captures and displays all request headers
- ✅ **Debug Endpoints** - Inspect captured headers and request history
- ✅ **MCP Protocol** - Implements basic MCP endpoints for testing
- ✅ **MCP Protocol** - Implements MCP endpoints (initialize, tools/list, tools/call)
- ✅ **Request Logging** - Tracks recent requests with timestamps
- ✅ **Self-Signed Certs** - Auto-generates certificates for HTTPS testing
- ✅ **Tool Execution** - Returns mock results for tool/call testing

## Quick Start

Expand All @@ -46,8 +47,11 @@ HTTPS: https://localhost:3001
Debug endpoints:
• /debug/headers - View captured headers
• /debug/requests - View request log
MCP endpoint:
• POST /mcp/v1/list_tools
MCP endpoints:
• POST with JSON-RPC (any path)
- method: "initialize"
- method: "tools/list"
- method: "tools/call"
======================================================================
Note: HTTPS uses a self-signed certificate (for testing only)
```
Expand Down Expand Up @@ -270,8 +274,9 @@ python dev-tools/mcp-mock-server/server.py 8080
This is a **development/testing tool only**:
- ❌ Not for production use
- ❌ No authentication/security
- ❌ Limited MCP protocol implementation
- ❌ Limited MCP protocol implementation (initialize, tools/list, tools/call only)
- ❌ Single-threaded (one request at a time)
- ❌ Mock responses only (not real tool execution)

For production, use real MCP servers.

183 changes: 130 additions & 53 deletions dev-tools/mcp-mock-server/server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,11 @@ def _capture_headers(self) -> None:
if len(request_log) > 10:
request_log.pop(0)

def do_POST(self) -> None: # pylint: disable=invalid-name
def do_POST(
self,
) -> (
None
): # pylint: disable=invalid-name,too-many-locals,too-many-branches,too-many-statements
"""Handle POST requests (MCP protocol endpoints)."""
self._capture_headers()

Expand All @@ -73,14 +77,24 @@ def do_POST(self) -> None: # pylint: disable=invalid-name
request_id = request_data.get("id", 1)
method = request_data.get("method", "unknown")
except (json.JSONDecodeError, UnicodeDecodeError):
request_data = {}
request_id = 1
method = "unknown"

# Log the RPC method in the request log
if request_log:
request_log[-1]["rpc_method"] = method

# Determine tool name based on authorization header to avoid collisions
auth_header = self.headers.get("Authorization", "")

# Initialize tool info defaults
tool_name = "mock_tool_no_auth"
tool_desc = "Mock tool with no authorization"
error_mode = False

# Match based on token content
match auth_header:
match True:
case _ if "test-secret-token" in auth_header:
tool_name = "mock_tool_file"
tool_desc = "Mock tool with file-based auth"
Expand All @@ -90,58 +104,116 @@ def do_POST(self) -> None: # pylint: disable=invalid-name
case _ if "my-client-token" in auth_header:
tool_name = "mock_tool_client"
tool_desc = "Mock tool with client-provided token"
case _ if "error-mode" in auth_header:
tool_name = "mock_tool_error"
tool_desc = "Mock tool configured to return errors"
error_mode = True
case _:
# No auth header or unrecognized token
tool_name = "mock_tool_no_auth"
tool_desc = "Mock tool with no authorization"

# Handle MCP protocol methods
if method == "initialize":
# Return MCP initialize response
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {
"protocolVersion": "2024-11-05",
"capabilities": {
"tools": {},
},
"serverInfo": {
"name": "mock-mcp-server",
"version": "1.0.0",
# Default case already set above
pass

# Log the tool name in the request log
if request_log:
request_log[-1]["tool_name"] = tool_name

# Handle MCP protocol methods using match statement
response: dict = {}
match method:
case "initialize":
# Return MCP initialize response
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {
"protocolVersion": "2024-11-05",
"capabilities": {
"tools": {},
},
"serverInfo": {
"name": "mock-mcp-server",
"version": "1.0.0",
},
},
},
}
elif method == "tools/list":
# Return list of tools with unique name
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {
"tools": [
{
"name": tool_name,
"description": tool_desc,
"inputSchema": {
"type": "object",
"properties": {
"message": {
"type": "string",
"description": "Test message",
}
}

case "tools/list":
# Return list of tools with unique name
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {
"tools": [
{
"name": tool_name,
"description": tool_desc,
"inputSchema": {
"type": "object",
"properties": {
"message": {
"type": "string",
"description": "Test message",
}
},
},
},
}
]
},
}
else:
# Generic success response for other methods
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {"status": "ok"},
}
}
]
},
}

case "tools/call":
# Handle tool execution
params = request_data.get("params", {})
tool_called = params.get("name", "unknown")
arguments = params.get("arguments", {})

# Check if error mode is enabled
if error_mode:
# Return error response
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {
"content": [
{
"type": "text",
"text": (
f"Error: Tool '{tool_called}' "
"execution failed - simulated error."
),
}
],
"isError": True,
},
}
else:
# Build result text
result_text = (
f"Mock tool '{tool_called}' executed successfully "
f"with arguments: {arguments}."
)

# Return successful tool execution result
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {
"content": [
{
"type": "text",
"text": result_text,
}
],
"isError": False,
},
}

case _:
# Generic success response for other methods
response = {
"jsonrpc": "2.0",
"id": request_id,
"result": {"status": "ok"},
}

self.send_response(200)
self.send_header("Content-Type", "application/json")
Expand All @@ -160,6 +232,11 @@ def do_GET(self) -> None: # pylint: disable=invalid-name
)
case "/debug/requests":
self._send_json_response(request_log)
case "/debug/clear":
# Clear the request log and last captured headers
request_log.clear()
last_headers.clear()
self._send_json_response({"status": "cleared", "request_count": 0})
case "/":
self._send_help_page()
case _:
Expand Down Expand Up @@ -273,10 +350,10 @@ def main() -> None:
https_port = http_port + 1

# Create HTTP server
http_server = HTTPServer(("", http_port), MCPMockHandler)
http_server = HTTPServer(("", http_port), MCPMockHandler) # type: ignore[arg-type]

# Create HTTPS server with self-signed certificate
https_server = HTTPServer(("", https_port), MCPMockHandler)
https_server = HTTPServer(("", https_port), MCPMockHandler) # type: ignore[arg-type]

# Generate or load self-signed certificate
script_dir = Path(__file__).parent
Expand Down
38 changes: 38 additions & 0 deletions dev-tools/test-configs/mcp-mock-test-noop.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Lightspeed Core Service - MCP Mock Server Test (Noop Auth)
service:
host: localhost
port: 8080
auth_enabled: false
workers: 1
color_log: true
access_log: true
llama_stack:
use_as_library_client: true
library_client_config_path: "dev-tools/test-configs/llama-stack-mcp-test.yaml"
user_data_collection:
feedback_enabled: false
transcripts_enabled: false
authentication:
module: "noop"
inference:
default_model: "gpt-4o-mini"
default_provider: "openai"
mcp_servers:
# Test 1: Static file-based authentication (HTTP)
- name: "mock-file-auth"
provider_id: "model-context-protocol"
url: "http://localhost:9000"
authorization_headers:
Authorization: "/tmp/lightspeed-mcp-test-token"
# Test 2: Kubernetes token forwarding (HTTP)
- name: "mock-k8s-auth"
provider_id: "model-context-protocol"
url: "http://localhost:9000"
authorization_headers:
Authorization: "kubernetes"
# Test 3: Client-provided token (HTTP - simplified for testing)
- name: "mock-client-auth"
provider_id: "model-context-protocol"
url: "http://localhost:9000"
authorization_headers:
Authorization: "client"
7 changes: 6 additions & 1 deletion docker-compose-library.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ services:
dockerfile: dev-tools/mcp-mock-server/Dockerfile
container_name: mcp-mock-server
ports:
- "3000:3000"
- "9000:3000"
networks:
- lightspeednet
healthcheck:
Expand Down Expand Up @@ -66,6 +66,11 @@ services:
- WATSONX_API_KEY=${WATSONX_API_KEY:-}
# Enable debug logging if needed
- LLAMA_STACK_LOGGING=${LLAMA_STACK_LOGGING:-}
entrypoint: >
/bin/bash -c "
printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
/app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
"
Comment on lines +69 to +73
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

find . -name "docker-compose-library.yaml" -o -name "*compose*.yaml" | head -20

Repository: lightspeed-core/lightspeed-stack

Length of output: 127

🏁 Script executed:

sed -n '60,80p' ./docker-compose-library.yaml

Repository: lightspeed-core/lightspeed-stack

Length of output: 1078

🌐 Web query:

bash PID1 signal handling Docker entrypoint

💡 Result:

When your Docker container’s PID 1 is bash (or /bin/sh -c), signal handling often surprises you:

  • PID 1 is special in Linux namespaces: if PID 1 doesn’t install handlers, some signals (notably SIGTERM/SIGINT) won’t have their normal “default action” of terminating the process. So docker stop may hang until Docker sends SIGKILL. [1]
  • If you use shell-form CMD/ENTRYPOINT (string form), Docker effectively runs your command under /bin/sh -c, and signals may not reach your real app (your app isn’t PID 1). Docker explicitly warns that shell-form ENTRYPOINT “does not pass signals”. [2]

Best practices for Docker entrypoints

  1. Prefer exec-form ENTRYPOINT / CMD (JSON array)
ENTRYPOINT ["/usr/local/bin/my-app"]
# or
CMD ["/usr/local/bin/my-app"]

This avoids /bin/sh -c wrappers and lets your app receive signals directly. [2]

  1. If you must use a shell script entrypoint, exec the long-running process
#!/bin/sh
set -e
# setup...
exec my-app "$@"

This replaces the shell with your app, so the app becomes PID 1 and receives SIGTERM. Docker’s own docs call out using exec for clean docker stop behavior. [2]

  1. Use a minimal init as PID 1 (recommended): --init (tini) or bake it in
  • Docker CLI: docker run --init … makes an init process run as PID 1 (backed by tini) and handle duties like signal forwarding and zombie reaping. [3]
  • Or explicitly: ENTRYPOINT ["tini","--","/your/entrypoint.sh"] (tini docs). [4]

Most common fix: switch to exec-form + ensure your entrypoint script ends with exec …, or run with --init.

Sources:
[1] Docker run reference (PID 1 signal behavior) https://www.docs4dev.com/docs/docker/19.03/engine/reference/run/
[2] Dockerfile reference (shell vs exec form; shell form doesn’t pass signals; use exec) https://docs.docker.com/reference/builder
[3] docker container run docs (--init, docker-init backed by tini) https://docs.docker.com/reference/cli/docker/container/run
[4] tini README (--init and usage) https://github.com/krallin/tini

🌐 Web query:

umask 077 file permissions /tmp default permissions bash

💡 Result:

  • umask 077 affects new files/directories your process creates anywhere, including under /tmp. It removes all group/other permissions from the requested mode. (sans.org)

    • Typical result:
      • New files: 0666 & ~0077 = 0600 (rw-------)
      • New directories: 0777 & ~0077 = 0700 (rwx------) (sans.org)

  • /tmp itself is normally world-writable with the sticky bit: mode 1777 (drwxrwxrwt). Your umask does not change /tmp’s existing permissions; it only affects what you create inside it. (linuxconfig.org)

  • Bash is not special here: if you set umask 077 in your shell (or in ~/.bashrc, etc.), then commands you run from that shell inherit it and will create /tmp files accordingly (unless the program explicitly chmods after creation or requests a more restrictive mode). (bs2manuals.ts.fujitsu.com)

  • Many systems ensure /tmp has mode 1777 at boot via systemd-tmpfiles rules (example entries show 1777 directories under /tmp). (freedesktop.org)

Citations:

🌐 Web query:

docker-compose entrypoint exec umask syntax YAML multiline string

💡 Result:

Use exec-form (entrypoint: as a YAML list) and run a shell explicitly so you can set umask and then exec the real process (so it becomes PID 1 and receives signals). Compose allows entrypoint as either a string or a list; the list form is the “Dockerfile-like”/exec style. [1]

services:
  app:
    image: your-image
    entrypoint:
      - /bin/sh
      - -c
      - |
          umask 002
          exec "$@"
      - --
    command: ["your-binary", "arg1", "arg2"]

Notes:

  • | is just YAML “multiline string”; it becomes the single script argument to sh -c (newlines included).
  • The -- is used as the $0 placeholder for sh -c; then command: becomes $@, which exec "$@" runs.
  • Using exec ... ensures the final process is PID 1 (important for signal handling), as recommended in Docker’s ENTRYPOINT docs. [2]

Sources: Docker Compose entrypoint reference [1]; Docker ENTRYPOINT shell vs exec form + exec for proper PID 1 behavior [2].

[1] (docs.docker.com)
[2] (docs.docker.com)

Citations:

Use exec and tighten token file permissions in the entrypoint.

Bash as PID1 prevents proper signal forwarding—docker stop may hang until SIGKILL. The token file in /tmp defaults to world-readable mode. Add umask 077 and exec the Python process to make it PID1 and restrict the token file to owner-only (0600).

🛠️ Proposed fix 
-    entrypoint: >
-      /bin/bash -c "
-      printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
-      /app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
-      "
+    entrypoint: >
+      /bin/bash -c "
+      umask 077
+      printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
+      exec /app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
+      "
🤖 Prompt for AI Agents 
In `@docker-compose-library.yaml` around lines 69 - 73, The entrypoint currently
starts a bash shell that writes the token and launches Python as a child;
replace that with a shell sequence that tightens token permissions and then uses
exec to replace PID 1 with the Python process: set umask 077 before creating the
token (or explicitly chmod the token to 0600 after printf to
/tmp/lightspeed-mcp-test-token) and then exec /app-root/.venv/bin/python3.12
/app-root/src/lightspeed_stack.py so the Python process becomes PID 1 and
receives signals correctly; update the entrypoint block that uses /bin/bash -c
and printf accordingly.

healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/liveness"]
interval: 10s # how often to run the check
Expand Down
7 changes: 6 additions & 1 deletion docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ services:
dockerfile: dev-tools/mcp-mock-server/Dockerfile
container_name: mcp-mock-server
ports:
- "3000:3000"
- "9000:3000"
networks:
- lightspeednet
healthcheck:
Expand Down Expand Up @@ -84,6 +84,11 @@ services:
- TENANT_ID=${TENANT_ID:-}
- CLIENT_ID=${CLIENT_ID:-}
- CLIENT_SECRET=${CLIENT_SECRET:-}
entrypoint: >
/bin/bash -c "
printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
/app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
"
Comment on lines +87 to +91
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n docker-compose.yaml | sed -n '85,95p'

Repository: lightspeed-core/lightspeed-stack

Length of output: 519

🌐 Web query:

bash PID1 signal handling swallow signals docker

💡 Result:

When you run a shell script as the container “main process”, bash becomes PID 1, and PID 1 has special signal semantics in Linux containers:

  • PID 1 doesn’t get the normal “default action” for many fatal signals (like SIGTERM): if PID 1 hasn’t installed a handler, the signal may effectively do nothing, so docker stop (SIGTERM) appears “swallowed”. This is why containers can linger until Docker eventually sends SIGKILL. [1], [2]
  • Even if bash traps/exits on SIGTERM, it typically won’t forward that signal to its child process(es) unless you explicitly do so—so your actual app keeps running. [3]

What to do instead (common fixes)

  1. Prefer making your app PID 1 (avoid bash as PID 1)
    Use Docker “exec form” so your process replaces the shell (no extra PID 1 shell layer). (This avoids the “shell doesn’t forward signals” problem.) [3]

  2. Add a minimal init as PID 1 (recommended for general robustness)
    Run with Docker’s built-in init (tini) so signals are forwarded and zombies are reaped: docker run --init … (or init: true in Compose). [4]

  3. If you must use bash as PID 1: trap + forward signals
    At minimum, install a SIGTERM trap (so PID 1 actually exits) and explicitly propagate termination to the child process group (otherwise children may ignore it). The need to trap TERM as PID 1 is the key point. [3]

Sources:
[1] https://stackoverflow.com/questions/71448484/why-does-foreground-job-ignore-job-control-signals-when-bash-is-running-as-pid-1
[2] https://semisignal.com/sigterm-and-pid1/
[3] https://cloud.theodo.com/en/blog/docker-processes-container
[4] https://raby.sh/sigterm-and-pid-1-why-does-a-container-linger-after-receiving-a-sigterm.html

🌐 Web query:

/tmp file permissions default linux umask security

💡 Result:

  • Default /tmp directory permissions on most Linux distros are drwxrwxrwt (octal 1777) owned by root:root: world-writable with the sticky bit so users can’t delete/rename other users’ files in /tmp even though everyone can create files there. This is commonly enforced at boot by systemd-tmpfiles rules (often D /tmp 1777 root root -). [1] [2]

  • umask controls the default permissions of new files/dirs created inside /tmp (it does not change /tmp’s own 1777 mode). When a program creates a file, the kernel applies:
    final_mode = requested_mode & ~umask (and similarly for directories). Typical default umask is 022, so a file created with requested mode 0666 becomes 0644. [3]

  • Security implications / best practice

    • /tmp is a shared namespace: predictable names like /tmp/foo can be pre-created by another user to cause failures or trick a program (DoS, symlink/hardlink-style attacks). Use unique, non-guessable names and safe creation patterns (e.g., “create-new” semantics), and prefer honoring $TMPDIR when set. [4]
    • The sticky bit does not prevent reads. If your program creates temp files with permissive modes (because it requests 0666 and relies on umask), other users may read them. Set sensitive temp files to owner-only permissions explicitly (and don’t rely on umask alone). [3]
    • For services, consider systemd PrivateTmp=yes to give the service its own private /tmp (and /var/tmp) namespace, reducing cross-process attacks via shared temp directories. [5]

  • Persistence note (/tmp vs /var/tmp)

    • /tmp is often cleared at boot and may be on tmpfs (RAM/swap), while /var/tmp is generally more persistent and intended to survive reboots. [6] [4]

Sources:
[1] https://www.baeldung.com/linux/sticky-bit
[2] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html
[3] https://manpages.debian.org/testing/manpages-dev/umask.2.en.html
[4] https://systemd.io/TEMPORARY_DIRECTORIES/
[5] https://www.redhat.com/de/blog/new-red-hat-enterprise-linux-7-security-feature-privatetmp
[6] https://unix.stackexchange.com/questions/30489/what-is-the-difference-between-tmp-and-var-tmp

Use exec and tighten token file permissions in the entrypoint.

Bash as PID 1 in containers won't forward SIGTERM to child processes and may ignore the signal itself without an explicit trap, preventing graceful shutdown. Additionally, the token file is created with default umask (typically 022), making it world-readable in the container. Apply umask 077 to restrict the token to owner-only permissions and use exec to make the Python process PID 1:

🛠️ Proposed fix 
-    entrypoint: >
-      /bin/bash -c "
-      printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
-      /app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
-      "
+    entrypoint: >
+      /bin/bash -c "
+      umask 077
+      printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
+      exec /app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
+      "
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
entrypoint: >
/bin/bash -c "
printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
/app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
"
entrypoint: >
/bin/bash -c "
umask 077
printf %s 'test-secret-token-123' > /tmp/lightspeed-mcp-test-token &&
exec /app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py
"
🤖 Prompt for AI Agents 
In `@docker-compose.yaml` around lines 87 - 91, The entrypoint should drop into
the Python process as PID 1 and create the token file with owner-only
permissions; update the entrypoint snippet that writes the token and launches
/app-root/.venv/bin/python3.12 /app-root/src/lightspeed_stack.py to first set
umask 077 (so the token file is created with owner-only perms) and then use exec
to replace the shell with the Python process (so the Python process becomes PID
1 and receives signals); locate the multline entrypoint block that prints
'test-secret-token-123' into /tmp/lightspeed-mcp-test-token and modify it
accordingly.

depends_on:
llama-stack:
condition: service_healthy
Expand Down
Loading
Loading