Skip to content

LCORE-598: RBAC E2E tests #1017

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
radofuchs merged 1 commit into from
Jan 22, 2026
Merged

LCORE-598: RBAC E2E tests #1017

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docker-compose-library.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,23 @@ services:
retries: 3 # how many times to retry before marking as unhealthy
start_period: 15s # time to wait before starting checks (increased for library initialization)

# Mock JWKS server for RBAC E2E tests
mock-jwks:
build:
context: ./tests/e2e/mock_jwks_server
dockerfile: Dockerfile
container_name: mock-jwks
ports:
- "8000:8000"
networks:
- lightspeednet
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 5s
timeout: 3s
retries: 3
start_period: 2s

networks:
lightspeednet:
driver: bridge
27 changes: 24 additions & 3 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,11 @@ services:
ports:
- "8321:8321" # Expose llama-stack on 8321 (adjust if needed)
volumes:
- ./run.yaml:/opt/app-root/run.yaml:Z
- ./run.yaml:/opt/app-root/run.yaml:z
- ${GCP_KEYS_PATH:-./tmp/.gcp-keys-dummy}:/opt/app-root/.gcp-keys:ro
- ./tests/e2e/rag:/opt/app-root/src/.llama/storage/rag:Z
- ./lightspeed-stack.yaml:/opt/app-root/lightspeed-stack.yaml:z
- ./lightspeed-stack.yaml:/opt/app-root/lightspeed-stack.yaml:ro
- llama-storage:/opt/app-root/src/.llama/storage
- ./tests/e2e/rag:/opt/app-root/src/.llama/storage/rag:z
environment:
- BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
Expand Down Expand Up @@ -97,6 +98,26 @@ services:
retries: 3 # how many times to retry before marking as unhealthy
start_period: 5s # time to wait before starting checks

# Mock JWKS server for RBAC E2E tests
mock-jwks:
build:
context: ./tests/e2e/mock_jwks_server
dockerfile: Dockerfile
container_name: mock-jwks
ports:
- "8000:8000"
networks:
- lightspeednet
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 5s
timeout: 3s
retries: 3
start_period: 2s
Comment on lines +101 to +116
Copy link
Contributor

@coderabbitai coderabbitai bot Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify if any service in docker-compose files should depend on mock-jwks for RBAC tests

echo "=== Checking docker-compose files for services that might need mock-jwks dependency ==="
rg -n "depends_on|mock-jwks" --type yaml -g 'docker-compose*.yaml'

echo ""
echo "=== Checking if RBAC config references the mock-jwks server ==="
fd -e yaml -e yml . tests/e2e/configuration --exec grep -l -i "jwk\|jwks\|8000" {} \; 2>/dev/null | head -5

Repository: lightspeed-core/lightspeed-stack

Length of output: 637

🏁 Script executed:

#!/bin/bash
# Get more context about the depends_on at line 87
echo "=== Context around line 87 in docker-compose.yaml ==="
sed -n '80,95p' docker-compose.yaml

echo ""
echo "=== Check all depends_on blocks in docker-compose.yaml ==="
sed -n '/depends_on:/,/^[^ ]/p' docker-compose.yaml | head -30

echo ""
echo "=== Search for mock-jwks in depends_on statements ==="
rg -B5 "mock-jwks" docker-compose.yaml | grep -A5 -B5 "depends_on"

Repository: lightspeed-core/lightspeed-stack

Length of output: 1767

🏁 Script executed:

#!/bin/bash
# Check RBAC configuration files to understand how mock-jwks is used
echo "=== Content of server-mode RBAC config ==="
head -50 tests/e2e/configuration/server-mode/lightspeed-stack-rbac.yaml

echo ""
echo "=== Search for JWKS/8000 references in RBAC config ==="
grep -n -i "jwk\|8000\|mock" tests/e2e/configuration/server-mode/lightspeed-stack-rbac.yaml

echo ""
echo "=== Check docker-compose.yaml to see which service runs RBAC tests ==="
rg -n "test|rbac|e2e" docker-compose.yaml | head -20

echo ""
echo "=== Check for any service that might need mock-jwks ==="
rg -B3 "lightspeed:" docker-compose.yaml | head -20

Repository: lightspeed-core/lightspeed-stack

Length of output: 2358

🏁 Script executed:

#!/bin/bash
# Identify which service has the depends_on at line 87
echo "=== Identify service with depends_on at line 87 ==="
sed -n '60,95p' docker-compose.yaml | head -40

echo ""
echo "=== Verify if lightspeed service currently depends on mock-jwks ==="
sed -n '/^  lightspeed:/,/^  [^ ]/p' docker-compose.yaml | grep -A10 "depends_on"

echo ""
echo "=== Check docker-compose-library.yaml for same pattern ==="
sed -n '/^  lightspeed:/,/^  [^ ]/p' docker-compose-library.yaml | grep -A10 "depends_on"

Repository: lightspeed-core/lightspeed-stack

Length of output: 1498

🏁 Script executed:

#!/bin/bash
# Get the complete lightspeed-stack service definition with depends_on
echo "=== Complete lightspeed-stack service definition ==="
sed -n '/^  lightspeed-stack:/,/^  [a-z]/p' docker-compose.yaml | head -50

echo ""
echo "=== Confirm mock-jwks is NOT in lightspeed-stack depends_on ==="
sed -n '/^  lightspeed-stack:/,/^  [a-z]/p' docker-compose.yaml | grep -c "mock-jwks"

echo ""
echo "=== Check if mock-jwks service has any depends_on ==="
sed -n '/^  mock-jwks:/,/^  [a-z]/p' docker-compose.yaml

Repository: lightspeed-core/lightspeed-stack

Length of output: 1717

🏁 Script executed:

#!/bin/bash
# Verify if lightspeed-stack service actually needs mock-jwks at startup
echo "=== Check lightspeed-stack.yaml config for RBAC/JWKS references ==="
grep -n -i "jwk\|mock-jwks" lightspeed-stack.yaml

echo ""
echo "=== Check if lightspeed-stack loads configuration at startup ==="
grep -n "load.*config\|read.*config\|auth" lightspeed-stack.yaml | head -10

echo ""
echo "=== Verify mock-jwks healthcheck reliability ==="
cat tests/e2e/mock_jwks_server/Dockerfile 2>/dev/null || echo "Dockerfile not found"

echo ""
echo "=== Check if there's a test file that documents RBAC test expectations ==="
find tests/e2e -name "*rbac*" -o -name "*auth*" | head -5

Repository: lightspeed-core/lightspeed-stack

Length of output: 789

🏁 Script executed:

#!/bin/bash
# Check how tests use the RBAC configuration
echo "=== Check if there's a test runner that manages config loading ==="
find tests/e2e -name "*.py" -o -name "*.sh" | xargs grep -l "lightspeed-stack-rbac" 2>/dev/null | head -3

echo ""
echo "=== Check if tests explicitly wait for mock-jwks ==="
grep -r "mock-jwks" tests/e2e --include="*.py" --include="*.feature" | head -10

echo ""
echo "=== Verify the actual architecture - is RBAC config loaded at startup or test-time? ==="
grep -n "load.*yaml\|load.*config" lightspeed/server.py 2>/dev/null | head -5 || echo "Checking main application startup..."
find . -name "main.py" -o -name "app.py" | xargs grep -l "config\|authentication" 2>/dev/null | head -3

Repository: lightspeed-core/lightspeed-stack

Length of output: 337

🏁 Script executed:

#!/bin/bash
# Check the test environment setup
echo "=== Test environment setup (environment.py) ==="
cat tests/e2e/features/environment.py

echo ""
echo "=== Check app startup to understand when config is loaded ==="
head -100 ./src/app/main.py

Repository: lightspeed-core/lightspeed-stack

Length of output: 14918

🏁 Script executed:

#!/bin/bash
# Final verification: confirm that config loading might fail if mock-jwks isn't ready
echo "=== Check if configuration loading validates JWKS endpoint during startup ==="
grep -r "jwk\|validate\|fetch" src/configuration --include="*.py" | head -10

echo ""
echo "=== Confirm there's no explicit wait for mock-jwks before config load in tests ==="
grep -n "mock-jwks\|8000" tests/e2e/features/environment.py

Repository: lightspeed-core/lightspeed-stack

Length of output: 305

🏁 Script executed:

#!/bin/bash
# Find configuration loading code
echo "=== Locate configuration files ==="
find . -name "configuration*" -type f | grep -E "\.py$" | head -10

echo ""
echo "=== Check lifespan context manager - does it wait for mock-jwks? ==="
grep -A10 "LIGHTSPEED_STACK_CONFIG_PATH" ./src/app/main.py

echo ""
echo "=== Final confirmation: mock-jwks NOT in lightspeed-stack depends_on ==="
sed -n '/^  lightspeed-stack:/,/^  [a-z]/p' docker-compose.yaml | grep -E "depends_on|mock-jwks"

Repository: lightspeed-core/lightspeed-stack

Length of output: 767

🏁 Script executed:

#!/bin/bash
# Check what configuration.load_configuration actually does
echo "=== Check configuration.load_configuration implementation ==="
sed -n '/def load_configuration/,/^def /p' ./src/configuration.py | head -50

echo ""
echo "=== Check if any JWKS validation happens at config load time ==="
grep -n "jwk\|fetch\|request\|validate" ./src/configuration.py

echo ""
echo "=== Verify docker-compose shows mock-jwks is independent ==="
grep -A15 "^  mock-jwks:" docker-compose.yaml | grep -E "depends_on|networks|healthcheck" -A2

Repository: lightspeed-core/lightspeed-stack

Length of output: 2372

Add mock-jwks to lightspeed-stack's depends_on to ensure RBAC test stability.

The RBAC E2E tests dynamically switch to a configuration that references http://mock-jwks:8000/.well-known/jwks.json at runtime (see environment.py:before_feature for RBAC tests). When the lightspeed-stack container restarts with this config, there's no guarantee that mock-jwks is healthy and ready to serve JWKS tokens. Without an explicit dependency, the first RBAC authentication request could fail if mock-jwks hasn't fully started.

Add the following to the lightspeed-stack service's depends_on section:

mock-jwks:
  condition: service_healthy

This ensures Docker enforces proper startup ordering for RBAC test scenarios.

🤖 Prompt for AI Agents 
In `@docker-compose.yaml` around lines 101 - 116, The lightspeed-stack service
needs an explicit startup dependency on the mock-jwks service to ensure the JWKS
server is healthy before RBAC tests run; update the lightspeed-stack service's
depends_on block to include mock-jwks with condition: service_healthy so Docker
will wait for mock-jwks' healthcheck to pass (mock-jwks is the service that
already defines a healthcheck).


volumes:
llama-storage:

networks:
lightspeednet:
driver: bridge
94 changes: 94 additions & 0 deletions tests/e2e/configuration/library-mode/lightspeed-stack-rbac.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
name: Lightspeed Core Service (RBAC E2E Tests - Library Mode)
service:
host: 0.0.0.0
port: 8080
auth_enabled: true
workers: 1
color_log: true
access_log: true

llama_stack:
use_as_library_client: true
library_client_config_path: run.yaml

user_data_collection:
feedback_enabled: true
feedback_storage: "/tmp/data/feedback"
transcripts_enabled: true
transcripts_storage: "/tmp/data/transcripts"

# Conversation cache for storing Q&A history
conversation_cache:
type: "sqlite"
sqlite:
db_path: "/tmp/data/conversation-cache.db"

# JWK token authentication with role extraction
authentication:
module: "jwk-token"
jwk_config:
url: "http://mock-jwks:8000/.well-known/jwks.json"
jwt_configuration:
user_id_claim: "sub"
username_claim: "name"
# Role rules: extract roles from JWT claims
role_rules:
# Grant 'admin' role to users with admin=true in JWT
- jsonpath: "$.admin"
operator: "equals"
value: [true]
roles: ["admin"]
# Grant 'user' role to users with role=user in JWT
- jsonpath: "$.role"
operator: "equals"
value: ["user"]
roles: ["user"]
# Grant 'viewer' role to users with role=viewer in JWT
- jsonpath: "$.role"
operator: "equals"
value: ["viewer"]
roles: ["viewer"]
# Grant 'query_only' role based on permissions array containing 'query'
- jsonpath: "$.permissions[*]"
operator: "contains"
value: "query"
roles: ["query_only"]

# Authorization: map roles to actions
authorization:
access_rules:
# Admin role gets full access
- role: "admin"
actions: ["admin"]
# User role can query, access conversations, and provide feedback
- role: "user"
actions:
- "query"
- "streaming_query"
- "get_conversation"
- "list_conversations"
- "delete_conversation"
- "update_conversation"
- "feedback"
- "get_models"
- "get_tools"
- "info"
- "model_override"
# Viewer role can only read (no mutations)
- role: "viewer"
actions:
- "get_conversation"
- "list_conversations"
- "get_models"
- "get_tools"
- "info"
# Query-only role can only query (no model_override - must use defaults)
- role: "query_only"
actions:
- "query"
- "streaming_query"
# Everyone (*) role gets basic info access
- role: "*"
actions:
- "info"

95 changes: 95 additions & 0 deletions tests/e2e/configuration/server-mode/lightspeed-stack-rbac.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
name: Lightspeed Core Service (RBAC E2E Tests)
service:
host: 0.0.0.0
port: 8080
auth_enabled: true
workers: 1
color_log: true
access_log: true

llama_stack:
use_as_library_client: false
url: http://llama-stack:8321
api_key: xyzzy

user_data_collection:
feedback_enabled: true
feedback_storage: "/tmp/data/feedback"
transcripts_enabled: true
transcripts_storage: "/tmp/data/transcripts"

# Conversation cache for storing Q&A history
conversation_cache:
type: "sqlite"
sqlite:
db_path: "/tmp/data/conversation-cache.db"

# JWK token authentication with role extraction
authentication:
module: "jwk-token"
jwk_config:
url: "http://mock-jwks:8000/.well-known/jwks.json"
jwt_configuration:
user_id_claim: "sub"
username_claim: "name"
# Role rules: extract roles from JWT claims
role_rules:
# Grant 'admin' role to users with admin=true in JWT
- jsonpath: "$.admin"
operator: "equals"
value: [true]
roles: ["admin"]
# Grant 'user' role to users with role=user in JWT
- jsonpath: "$.role"
operator: "equals"
value: ["user"]
roles: ["user"]
# Grant 'viewer' role to users with role=viewer in JWT
- jsonpath: "$.role"
operator: "equals"
value: ["viewer"]
roles: ["viewer"]
# Grant 'query_only' role based on permissions array containing 'query'
- jsonpath: "$.permissions[*]"
operator: "contains"
value: "query"
roles: ["query_only"]

# Authorization: map roles to actions
authorization:
access_rules:
# Admin role gets full access
- role: "admin"
actions: ["admin"]
# User role can query, access conversations, and provide feedback
- role: "user"
actions:
- "query"
- "streaming_query"
- "get_conversation"
- "list_conversations"
- "delete_conversation"
- "update_conversation"
- "feedback"
- "get_models"
- "get_tools"
- "info"
- "model_override"
# Viewer role can only read (no mutations)
- role: "viewer"
actions:
- "get_conversation"
- "list_conversations"
- "get_models"
- "get_tools"
- "info"
# Query-only role can only query (no model_override - must use defaults)
- role: "query_only"
actions:
- "query"
- "streaming_query"
# Everyone (*) role gets basic info access
- role: "*"
actions:
- "info"

14 changes: 14 additions & 0 deletions tests/e2e/features/environment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,15 @@ def before_feature(context: Context, feature: Feature) -> None:
switch_config(context.feature_config)
restart_container("lightspeed-stack")

if "RBAC" in feature.tags:
mode_dir = "library-mode" if context.is_library_mode else "server-mode"
context.feature_config = (
f"tests/e2e/configuration/{mode_dir}/lightspeed-stack-rbac.yaml"
)
context.default_config_backup = create_config_backup("lightspeed-stack.yaml")
switch_config(context.feature_config)
restart_container("lightspeed-stack")

if "Feedback" in feature.tags:
context.hostname = os.getenv("E2E_LSC_HOSTNAME", "localhost")
context.port = os.getenv("E2E_LSC_PORT", "8080")
Expand All @@ -259,6 +268,11 @@ def after_feature(context: Context, feature: Feature) -> None:
restart_container("lightspeed-stack")
remove_config_backup(context.default_config_backup)

if "RBAC" in feature.tags:
switch_config(context.default_config_backup)
restart_container("lightspeed-stack")
remove_config_backup(context.default_config_backup)

if "Feedback" in feature.tags:
for conversation_id in context.feedback_conversations:
url = f"http://{context.hostname}:{context.port}/v1/conversations/{conversation_id}"
Expand Down
2 changes: 2 additions & 0 deletions tests/e2e/features/info.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ Feature: Info tests
{"detail": {"response": "Unable to connect to Llama Stack", "cause": "Connection error."}}
"""

#https://issues.redhat.com/browse/LCORE-1211
@skip
Scenario: Check if tools endpoint is working
Given The system is in default state
When I access REST API endpoint "tools" using HTTP GET method
Expand Down
2 changes: 2 additions & 0 deletions tests/e2e/features/query.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ Feature: Query endpoint API tests
| Fragments in LLM response |
| checkout |

#enable on demand
@skip
Scenario: Check if LLM ignores new system prompt in same conversation
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
Expand Down
Loading
Loading