|
| 1 | +# OpenAI Model Provider Integration Testing |
| 2 | + |
| 3 | +This module includes comprehensive integration tests for the OpenAI model provider using WireMock in proxy/replay mode. This allows testing against both real OpenAI APIs and recorded responses. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The integration tests can operate in two modes: |
| 8 | + |
| 9 | +1. **Real API Mode**: Tests run against the actual OpenAI API |
| 10 | +2. **WireMock Mode**: Tests run against recorded API responses |
| 11 | + |
| 12 | +The mode is automatically determined based on the `ANTHROPIC_API_KEY` environment variable: |
| 13 | +- Real API: When `ANTHROPIC_API_KEY` does NOT start with "test" |
| 14 | +- WireMock: When `ANTHROPIC_API_KEY` starts with "test" or is not set |
| 15 | + |
| 16 | +## Cost-Effective Testing |
| 17 | + |
| 18 | +The tests use the most cost-effective OpenAI models: |
| 19 | +- **Embedding Model**: `text-embedding-3-small` (cheaper than text-embedding-3-large) |
| 20 | +- **Chat Model**: `gpt-3.5-turbo` (significantly cheaper than GPT-4 models) |
| 21 | + |
| 22 | +## Setup and Usage |
| 23 | + |
| 24 | +### 1. Running Tests Against Recorded Responses (Default) |
| 25 | + |
| 26 | +This is the recommended approach for CI/CD and regular development: |
| 27 | + |
| 28 | +```bash |
| 29 | +# Set test API key (or don't set it at all) |
| 30 | +export ANTHROPIC_API_KEY=test-key |
| 31 | + |
| 32 | +# Run the integration tests |
| 33 | +mvn test -Dtest=AnthropicModelProviderIT |
| 34 | +``` |
| 35 | + |
| 36 | +### 2. Capturing New API Responses |
| 37 | + |
| 38 | +When you need to update the recorded responses or test new functionality: |
| 39 | + |
| 40 | +#### Step 1: Start WireMock Proxy |
| 41 | +```bash |
| 42 | +# Set your real OpenAI API key |
| 43 | +export ANTHROPIC_API_KEY=your-real-api-key |
| 44 | + |
| 45 | +# Start WireMock proxy |
| 46 | +mvn exec:java@wiremock-proxy |
| 47 | +``` |
| 48 | + |
| 49 | +#### Step 2: Run Tests Through Proxy |
| 50 | +In another terminal: |
| 51 | +```bash |
| 52 | +# Keep the real API key set |
| 53 | +export ANTHROPIC_API_KEY=your-real-api-key |
| 54 | + |
| 55 | +# Point tests to WireMock proxy |
| 56 | +export ANTHROPIC_BASE_URL=http://localhost:8089/v1 |
| 57 | + |
| 58 | +# Run tests to capture responses |
| 59 | +mvn test -Dtest=AnthropicModelProviderIT -nsu |
| 60 | +``` |
| 61 | + |
| 62 | +#### Step 3: Stop Proxy and Verify |
| 63 | +- Stop the WireMock proxy (Ctrl+C) |
| 64 | +- Check that new mapping files were created in `target/wiremock/mappings/` |
| 65 | +- Check that response files were created in `target/wiremock/__files/` |
| 66 | +- Copy mapping and response files as needed to `test/resources/wiremock` |
| 67 | + |
| 68 | +### 3. Running Tests Against Real API |
| 69 | + |
| 70 | +For validation or testing new functionality: |
| 71 | + |
| 72 | +```bash |
| 73 | +# Set your real OpenAI API key (must NOT start with "test") |
| 74 | +export ANTHROPIC_API_KEY=sk-your-real-api-key |
| 75 | + |
| 76 | +# Run tests against real API |
| 77 | +mvn test -Dtest=AnthropicModelProviderIntegrationTest |
| 78 | +``` |
| 79 | + |
| 80 | +## Directory Structure |
| 81 | + |
| 82 | +``` |
| 83 | +src/test/resources/wiremock/ |
| 84 | +├── mappings/ # Request/response mappings |
| 85 | +│ ├── chat-completion-request.json # Chat completion API mapping |
| 86 | +│ └── streaming-chat-completion-request.json # Streaming Chat completion API mapping |
| 87 | +└── __files/ # Response body files |
| 88 | + ├── chat-completion-response.json # Example chat completion response |
| 89 | + └── streaming-chat-completion-response.txt # Example streaming chat completion response |
| 90 | +``` |
| 91 | + |
| 92 | +## Test Coverage |
| 93 | + |
| 94 | +The integration tests cover: |
| 95 | + |
| 96 | +- **Chat Model**: Conversation handling with response validation |
| 97 | +- **Streaming Chat Model**: Real-time conversation with streaming responses |
| 98 | + |
| 99 | +## Troubleshooting |
| 100 | + |
| 101 | +### WireMock Proxy Not Starting |
| 102 | +- Ensure port 8089 is available |
| 103 | +- Check that Java is installed and accessible |
| 104 | + |
| 105 | +### Tests Failing in WireMock Mode |
| 106 | +- Ensure recorded responses exist in `src/test/resources/wiremock/` |
| 107 | +- Check that mapping files match the expected request patterns |
| 108 | +- Verify response files contain valid JSON or SSE responses (for streaming chat) |
| 109 | + |
| 110 | +### Tests Failing in Real API Mode |
| 111 | +- Verify your OpenAI API key is valid and has sufficient credits |
| 112 | +- Check internet connectivity |
| 113 | +- Ensure you're not hitting rate limits |
| 114 | + |
| 115 | +### API Key Issues |
| 116 | +- Real API keys must start with `sk-` |
| 117 | +- Test API keys should start with `test` to trigger WireMock mode |
| 118 | +- Never commit real API keys to version control |
| 119 | + |
| 120 | +## Security Notes |
| 121 | + |
| 122 | +- **Never commit real API keys** to version control |
| 123 | +- Use test keys for recorded responses |
| 124 | +- The example responses contain only synthetic data |
| 125 | +- Real API responses may contain sensitive information - review before committing |
| 126 | + |
| 127 | +## Cost Management |
| 128 | + |
| 129 | +When testing against the real API: |
| 130 | +- Tests use the cheapest available models |
| 131 | +- Each test run costs approximately $0.01-0.02 USD |
| 132 | +- Monitor your OpenAI usage dashboard |
| 133 | +- Consider using lower limits on your API key for testing |
0 commit comments