-
Notifications
You must be signed in to change notification settings - Fork 46
docs(sdk): document ask_oracle tool #566
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
689f391
docs(sdk): document ask oracle tool
enyst f380da2
docs(sdk): clarify oracle prompt context
openhands-agent 1619114
Update ask oracle example path
enyst 8bd734d
docs(sdk): align ask_oracle guide with convention-based design
openhands-agent b3b8bb7
Merge remote-tracking branch 'origin/main' into gpt/docs-pr566
enyst c919bd0
docs(sdk): clarify Ask Oracle setup
enyst f6baea5
docs(sdk): link Oracle usage metrics
enyst a7601ad
docs(sdk): address Ask Oracle review
enyst File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,180 @@ | ||
| --- | ||
| title: Ask Oracle | ||
| description: Let an agent consult a saved Oracle LLM profile for stateless second-opinion advice. | ||
| --- | ||
|
|
||
| import RunExampleCode from "/sdk/shared-snippets/how-to-run-example.mdx"; | ||
|
|
||
| > A ready-to-run example is available [here](#ready-to-run-example)! | ||
|
|
||
| Use `ask_oracle` when an agent should consult a stronger or more specialized | ||
| model for a second opinion without switching its active model. | ||
|
|
||
| ## When to Use It | ||
|
|
||
| `ask_oracle` is useful when an agent is: | ||
|
|
||
| - Stuck or uncertain about its next step | ||
| - Comparing implementation approaches | ||
| - Reviewing a risky or difficult decision | ||
| - Asked by the user to get a second opinion | ||
|
|
||
| ## How It Works | ||
|
|
||
| When the agent calls `ask_oracle`: | ||
|
|
||
| 1. The tool loads the saved LLM profile named `oracle`. | ||
| 2. The Oracle receives a dedicated system prompt and a user message containing | ||
| the agent's question and optional context. | ||
| 3. The Oracle returns a text recommendation to the original agent. | ||
| 4. The original agent continues the conversation with its existing model. | ||
|
|
||
| The Oracle does not receive the conversation history or any tools. It cannot | ||
| modify the workspace directly. Its token usage and cost are included in the | ||
| conversation's combined metrics. | ||
|
|
||
| <Note> | ||
| The tool does not fall back to the agent's active model. If the `oracle` | ||
| profile is missing or cannot be loaded, the tool returns an error observation | ||
| telling the agent that the Oracle is unavailable. | ||
| </Note> | ||
|
|
||
| ## Configure the Oracle Profile | ||
|
|
||
| The tool resolves its model by convention from a saved LLM profile named | ||
| `oracle`. There is no dedicated agent setting for selecting another profile. | ||
|
|
||
| To enable it: | ||
|
|
||
| 1. Save a usable LLM configuration under the name `oracle`. See | ||
| [LLM Profile Store](/sdk/guides/llm-profile-store). | ||
| 2. Add `AskOracleTool` to the agent's tools: | ||
|
|
||
| ```python icon="python" wrap focus={2, 5} | ||
| from openhands.sdk import Agent, Tool | ||
| from openhands.tools.ask_oracle import AskOracleTool | ||
|
|
||
| agent = Agent( | ||
| llm=primary_llm, | ||
| tools=[Tool(name=AskOracleTool.name)], | ||
| ) | ||
| ``` | ||
|
|
||
| By default, `LocalConversation` reads profiles from | ||
| `~/.openhands/profiles`. If you use a custom profile directory, pass the same | ||
| directory to both `LLMProfileStore` and `LocalConversation` through | ||
| `profile_store_dir`. | ||
|
|
||
| <Warning> | ||
| Do not place literal API keys in source code. The ready-to-run example reads | ||
| its key from the environment and stores the Oracle profile in a temporary | ||
| directory, which is removed after the example exits. Follow the | ||
| [LLM Profile Store](/sdk/guides/llm-profile-store) guidance when creating a | ||
| persistent profile. | ||
| </Warning> | ||
|
|
||
| ## Ask Oracle vs. Switch LLM | ||
|
|
||
| `ask_oracle` makes one stateless call to another model and then returns control | ||
| to the original agent. It never changes the active conversation model. | ||
|
|
||
| Use `switch_profile()` or the `switch_llm` tool instead when subsequent agent | ||
| turns should run on a different saved profile. See | ||
| [LLM Profile Store](/sdk/guides/llm-profile-store#mid-conversation-model-switching). | ||
|
|
||
| ## Ready-to-run Example | ||
|
|
||
| <Note> | ||
| This example is available on GitHub: [examples/01_standalone_sdk/58_ask_oracle_tool/main.py](https://github.com/OpenHands/software-agent-sdk/blob/main/examples/01_standalone_sdk/58_ask_oracle_tool/main.py) | ||
| </Note> | ||
|
|
||
| ```python icon="python" expandable examples/01_standalone_sdk/58_ask_oracle_tool/main.py | ||
| """Consult the Oracle end-to-end with the ask_oracle tool. | ||
|
|
||
| The Oracle is a saved LLM profile resolved by convention under the name | ||
| ``oracle``. This example wires two profiles — the agent's primary model and a | ||
| separate ``oracle`` model — adds ``Tool(name="ask_oracle")`` to the agent, then | ||
| drives a normal conversation: the agent decides to call ``ask_oracle``, the tool | ||
| consults the ``oracle`` profile, and the agent uses the Oracle's answer to reply. | ||
|
|
||
| Usage: | ||
| LLM_API_KEY=... LLM_BASE_URL=https://llm-proxy.app.all-hands.dev \ | ||
| uv run python examples/01_standalone_sdk/58_ask_oracle_tool/main.py | ||
|
|
||
| Note: | ||
| The example saves the ``oracle`` profile in a temporary directory so it | ||
| does not modify the user's default profile store. | ||
| """ | ||
|
|
||
| import os | ||
| import tempfile | ||
|
|
||
| from pydantic import SecretStr | ||
|
|
||
| from openhands.sdk import LLM, Agent, LocalConversation, Tool | ||
| from openhands.sdk.llm.llm_profile_store import LLMProfileStore | ||
| from openhands.tools.ask_oracle import ORACLE_PROFILE_NAME, AskOracleTool | ||
|
|
||
|
|
||
| DEFAULT_BASE_URL = "https://llm-proxy.app.all-hands.dev" | ||
| # The agent's primary model (follows the standard LLM_MODEL env like other | ||
| # examples). The Oracle defaults to the same model; override ASK_ORACLE_MODEL to | ||
| # point the "oracle" profile at a different/stronger model. | ||
| PRIMARY_MODEL = os.getenv("ASK_ORACLE_PRIMARY_MODEL") or os.getenv( | ||
| "LLM_MODEL", "openai/gpt-5.5" | ||
| ) | ||
| ORACLE_MODEL = os.getenv("ASK_ORACLE_MODEL", PRIMARY_MODEL) | ||
|
|
||
| api_key = os.getenv("LLM_API_KEY") | ||
| assert api_key is not None, "LLM_API_KEY environment variable is not set." | ||
| base_url = os.getenv("LLM_BASE_URL", DEFAULT_BASE_URL) | ||
|
|
||
| with tempfile.TemporaryDirectory() as profile_store_dir: | ||
| store = LLMProfileStore(profile_store_dir) | ||
| store.save( | ||
| ORACLE_PROFILE_NAME, | ||
| LLM( | ||
| model=ORACLE_MODEL, | ||
| api_key=SecretStr(api_key), | ||
| base_url=base_url, | ||
| usage_id="oracle", | ||
| ), | ||
| include_secrets=True, | ||
| ) | ||
|
|
||
| primary_llm = LLM( | ||
| model=PRIMARY_MODEL, | ||
| api_key=SecretStr(api_key), | ||
| base_url=base_url, | ||
| usage_id="primary", | ||
| ) | ||
| agent = Agent(llm=primary_llm, tools=[Tool(name=AskOracleTool.name)]) | ||
| conversation = LocalConversation( | ||
| agent=agent, | ||
| workspace=os.getcwd(), | ||
| profile_store_dir=profile_store_dir, | ||
| ) | ||
|
|
||
| print(f"Primary model: {conversation.agent.llm.model}") | ||
| print(f"Oracle model: {ORACLE_MODEL}") | ||
| conversation.send_message( | ||
| "Call the oracle to ask it for its opinion on the weather today, " | ||
| "then just tell me in two words how it's like." | ||
| ) | ||
| conversation.run() | ||
|
|
||
| cost = conversation.conversation_stats.get_combined_metrics().accumulated_cost | ||
| print(f"Total cost: ${cost:.6f}") | ||
| print(f"EXAMPLE_COST: {cost}") | ||
| ``` | ||
|
|
||
| <RunExampleCode path_to_script="examples/01_standalone_sdk/58_ask_oracle_tool/main.py"/> | ||
|
|
||
| ## Next Steps | ||
|
|
||
| - **[LLM Profile Store](/sdk/guides/llm-profile-store)** - Create and manage | ||
| reusable LLM configurations | ||
| - **[LLM Metrics](/sdk/guides/metrics)** - Track usage and cost across the | ||
| primary and Oracle models | ||
| - **[Custom Tools](/sdk/guides/custom-tools)** - Build tools with custom | ||
| behavior | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Suggestion:
focus={2, 5}highlights lines 2 and 5 of the code block, but line 5 isllm=primary_llm,— unrelated toAskOracleTool. The key line to emphasize is line 6 (tools=[Tool(name=AskOracleTool.name)],).Change to
focus={2, 6}so readers see the import (line 2) and thetools=argument (line 6) highlighted together, which is what this section is teaching.