All AI goes through apps/api/app/services/llm/. Business code builds a request and calls
LLMService; only providers/ imports a vendor SDK (a test enforces this).
- Create an API key in Google AI Studio.
- Pick a current text model and embedding model from Google's model list (https://ai.google.dev/gemini-api/docs/models). Names change often, which is why there is no default in code.
- Put them in the repo-root
.envand restart the API:
LLM_PROVIDER=gemini
GEMINI_API_KEY=...
GEMINI_MODEL=...
GEMINI_EMBEDDING_MODEL=...
- Open an organization page. As an admin you'll see the AI section; click Test AI connection. It sends one tiny prompt and shows latency and tokens.
The API key stays on the server. The browser never sees it, and the status endpoint only reports setting names that are missing, never values.
Set LLM_PROVIDER=anthropic, ANTHROPIC_API_KEY and ANTHROPIC_MODEL. Anthropic has no
embeddings API, so embeddings stay on Gemini (GEMINI_API_KEY + GEMINI_EMBEDDING_MODEL).
Using Claude Code to develop this project needs no Anthropic key here.
| Need | Call |
|---|---|
| Plain answer | generate(request, ctx) |
| JSON validated against a Pydantic model (one repair attempt) | generate_structured(request, Model, ctx) |
| Model calls our read-only tools, bounded steps | run_tools(request, [Tool(...)], ctx) |
| Embeddings, batched | embed(texts, "document" | "query", ctx) |
Built in: per-call timeout (LLM_TIMEOUT_SECONDS), retries with backoff on rate limits,
timeouts and 5xx (LLM_MAX_ATTEMPTS), and one llm_usage row per operation
(provider, model, tokens, latency, ok/error). Prompts and replies are never stored.
A spent daily quota (for example the free tier's small daily request limit) is different from a
short rate limit: waiting minutes can't fix it, so it is not retried. Reviews, CI analyses and
indexing fail at once with a clear message, and Ask and the connection test return a 429 that says
the same, including roughly when it resets if Google says. Fix it by waiting, turning on billing,
or switching GEMINI_MODEL to a model with a bigger allowance.
The AI usage panel on the organization page shows calls, tokens and errors per feature and
model (docs/USAGE.md).
- Anything from a repository, PR, issue or CI log goes through
untrusted(label, text)and the system prompt includesUNTRUSTED_CONTENT_NOTICE. - Tool handlers are our code. Model-supplied arguments are untrusted: validate them, keep handlers read-only and bounded.
- Nothing the model returns is ever executed or posted to GitHub without a human approving it.
Repository indexing and code search (docs/INDEXING.md) embed code with GEMINI_EMBEDDING_MODEL. Questions about code (docs/ASK.md) and pull request reviews (docs/REVIEWS.md) use GEMINI_MODEL.
Each embedding batch is recorded in llm_usage (feature indexing, code_search, ask_retrieval, ask or review).
| Setting | Default | Meaning |
|---|---|---|
LLM_PROVIDER |
gemini |
gemini or anthropic |
LLM_TIMEOUT_SECONDS |
60 | per provider call |
LLM_MAX_ATTEMPTS |
3 | calls per operation, retries included |
LLM_MAX_OUTPUT_TOKENS |
4096 | default cap when a caller doesn't set one |
EMBEDDING_DIMENSIONS |
768 | must match the pgvector column (added with indexing) |
WEBHOOK_DELIVERY_RETENTION_DAYS |
7 | how long GitHub delivery ids are remembered |