Adding Native Support for LLM Provider Tools in the Playground (e.g. Gemini Search Grounding)

[mmabrouk]

Context

Many LLM providers now offer pre-built tools through their APIs that enhance model capabilities. For example:

• Google's Gemini offers structured output, code execution, function calling, and grounded Google Search
• OpenAI provides web search, file search, and computer use through their Response API
• Claude offers web_search tool

You can see Gemini's implementation in Google AI Studio.

Proposal

We should add native support for provider-specific tools within the Agenta Playground and comparison service. This would give users access to powerful capabilities like grounded search without requiring custom integration work.

The LiteLLM library already supports many of these tools, including Google Search for Gemini, which should make implementation relatively straightforward.

Implementation Considerations

Frontend/UX Challenges

• How do we differentiate between custom JSON function tools and provider-specific pre-built tools?
• How do we handle the varying capabilities across different providers?

Would love to hear any thoughts on this idea. Please upvote if you think this would be a valuable.

[mmabrouk]

Shipped