feat(sdk): add ask_oracle tool

[enyst]

HUMAN:
This PR proposes an Oracle tool, for the agent to ask a more capable LLM when it encounters a difficulty, when it needs a second opinion, or when the user tells it to.

• A human has tested these changes.

AGENT:

📄 Visual walkthrough for reviewers: How ask_oracle works — a second opinion without switching your model — a code-grounded show-me: mechanism trace, and why it's not switch_llm and not a subagent.

Why

Agents sometimes need a second opinion from a stronger or more specialized saved LLM profile without permanently switching the active conversation profile. This adds a minimal ask_oracle tool powered by an oracle profile name so the agent can consult that Oracle profile statelessly, get a response, and continue with its current LLM.

Since the tool is defined in the tools package, we should probably not make a setting in the SDK for it. Instead, if users define the oracle LLM profile, it will be used for the Oracle tool.

Execution model

ask_oracle follows the existing synchronous tool-executor pattern: it implements __call__ and uses make_llm_completion. When LocalConversation.arun() executes the tool, synchronous executors run in the thread pool, so the Oracle request does not block the event loop. This is the same pattern used by the sibling vision_inspect and switch_llm tools. A native __acall__/amake_llm_completion path could avoid occupying a worker for a high-latency request, but this PR stays consistent with those tools; async support can be added across the auxiliary LLM tools together if it becomes necessary.

Summary

• Add an ask_oracle tool in openhands-tools that asks a saved Oracle LLM profile (resolved by convention under the name oracle) for stateless second-opinion guidance.
• Add unit coverage and a runnable example.
• Allow LocalConversation to use a custom LLM profile directory so the example can isolate its temporary Oracle profile.

Closes #3672.

Documentation: OpenHands/docs#566

REST API contract changes

Compared with base OpenAPI 90917f02ab23 for public /api/** paths.

--- base public OpenAPI
+++ head public OpenAPI
@@ -701 +701 @@
-schema Action oneOf=[MCPToolAction,FinishAction,InvokeSkillAction,SwitchLLMAction,ThinkAction,VisionInspectAction,BrowserAction,BrowserClickAction,BrowserCloseTabAction,BrowserGetContentAction,BrowserGetStateAction,BrowserGetStorageAction,BrowserGoBackAction,BrowserListTabsAction,BrowserNavigateAction,BrowserScrollAction,BrowserSetStorageAction,BrowserStartRecordingAction,BrowserStopRecordingAction,BrowserSwitchTabAction,BrowserTypeAction,DelegateAction,FileEditorAction,EditAction,ListDirectoryAction,ReadFileAction,WriteFileAction,GlobAction,GrepAction,PlanningFileEditorAction,TaskAction,TaskTrackerAction,TerminalAction,WorkflowAction]
+schema Action oneOf=[MCPToolAction,FinishAction,InvokeSkillAction,SwitchLLMAction,ThinkAction,VisionInspectAction,AskOracleAction,BrowserAction,BrowserClickAction,BrowserCloseTabAction,BrowserGetContentAction,BrowserGetStateAction,BrowserGetStorageAction,BrowserGoBackAction,BrowserListTabsAction,BrowserNavigateAction,BrowserScrollAction,BrowserSetStorageAction,BrowserStartRecordingAction,BrowserStopRecordingAction,BrowserSwitchTabAction,BrowserTypeAction,DelegateAction,FileEditorAction,EditAction,ListDirectoryAction,ReadFileAction,WriteFileAction,GlobAction,GrepAction,PlanningFileEditorAction,TaskAction,TaskTrackerAction,TerminalAction,WorkflowAction]
@@ -871,0 +872,16 @@
+schema AskOracleAction property context optional schema=anyOf=[type="string",type="null"]
+schema AskOracleAction property kind required schema=type="string" const="AskOracleAction"
+schema AskOracleAction property question required schema=type="string"
+schema AskOracleAction type="object" additionalProperties=false
+schema AskOracleObservation property content optional schema=type="array" items=anyOf=[TextContent,ImageContent]
+schema AskOracleObservation property is_error optional schema=type="boolean" default=false
+schema AskOracleObservation property kind required schema=type="string" const="AskOracleObservation"
+schema AskOracleObservation type="object" additionalProperties=false
+schema AskOracleTool property action_type required schema=type="string"
+schema AskOracleTool property annotations optional schema=anyOf=[openhands__sdk__tool__tool__ToolAnnotations,type="null"]
+schema AskOracleTool property description required schema=type="string"
+schema AskOracleTool property kind required schema=type="string" const="AskOracleTool"
+schema AskOracleTool property meta optional schema=anyOf=[type="object" additionalProperties=true,type="null"]
+schema AskOracleTool property observation_type optional schema=anyOf=[type="string",type="null"]
+schema AskOracleTool property title required schema=type="string"
+schema AskOracleTool type="object"
@@ -2035 +2051 @@
-schema Observation oneOf=[MCPToolObservation,FinishObservation,InvokeSkillObservation,SwitchLLMObservation,ThinkObservation,VisionInspectObservation,ClientToolObservation,BrowserObservation,DelegateObservation,FileEditorObservation,EditObservation,ListDirectoryObservation,ReadFileObservation,WriteFileObservation,GlobObservation,GrepObservation,PlanningFileEditorObservation,TaskObservation,TaskTrackerObservation,TerminalObservation,WorkflowObservation]
+schema Observation oneOf=[MCPToolObservation,FinishObservation,InvokeSkillObservation,SwitchLLMObservation,ThinkObservation,VisionInspectObservation,ClientToolObservation,AskOracleObservation,BrowserObservation,DelegateObservation,FileEditorObservation,EditObservation,ListDirectoryObservation,ReadFileObservation,WriteFileObservation,GlobObservation,GrepObservation,PlanningFileEditorObservation,TaskObservation,TaskTrackerObservation,TerminalObservation,WorkflowObservation]
@@ -2622 +2638 @@
-schema ToolDefinition oneOf=[MCPToolDefinition,FinishTool,InvokeSkillTool,SwitchLLMTool,ThinkTool,VisionInspectTool,ClientTool,BrowserClickTool,BrowserCloseTabTool,BrowserGetContentTool,BrowserGetStateTool,BrowserGetStorageTool,BrowserGoBackTool,BrowserListTabsTool,BrowserNavigateTool,BrowserScrollTool,BrowserSetStorageTool,BrowserStartRecordingTool,BrowserStopRecordingTool,BrowserSwitchTabTool,BrowserToolSet,BrowserTypeTool,FileEditorTool,EditTool,ListDirectoryTool,ReadFileTool,WriteFileTool,GlobTool,GrepTool,PlanningFileEditorTool,TaskTool,TaskToolSet,TaskTrackerTool,TerminalTool,WorkflowTool,WorkflowToolSet]
+schema ToolDefinition oneOf=[MCPToolDefinition,FinishTool,InvokeSkillTool,SwitchLLMTool,ThinkTool,VisionInspectTool,ClientTool,AskOracleTool,BrowserClickTool,BrowserCloseTabTool,BrowserGetContentTool,BrowserGetStateTool,BrowserGetStorageTool,BrowserGoBackTool,BrowserListTabsTool,BrowserNavigateTool,BrowserScrollTool,BrowserSetStorageTool,BrowserStartRecordingTool,BrowserStopRecordingTool,BrowserSwitchTabTool,BrowserToolSet,BrowserTypeTool,FileEditorTool,EditTool,ListDirectoryTool,ReadFileTool,WriteFileTool,GlobTool,GrepTool,PlanningFileEditorTool,TaskTool,TaskToolSet,TaskTrackerTool,TerminalTool,WorkflowTool,WorkflowToolSet]

Validation

The five-model integration run completed successfully at 43/44 applicable scenarios (97.7%). The sole Claude miss was an unrelated executable-bit omission in t02_add_bash_hello; a targeted rerun passed 1/1.

How to Test

• uv run pre-commit run --files openhands-sdk/openhands/sdk/tool/builtins/__init__.py openhands-tools/openhands/tools/__init__.py openhands-tools/openhands/tools/ask_oracle/__init__.py openhands-tools/openhands/tools/ask_oracle/definition.py openhands-tools/openhands/tools/ask_oracle/impl.py tests/tools/ask_oracle/test_ask_oracle.py openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py tests/sdk/conversation/test_switch_model.py tests/examples/test_examples.py examples/01_standalone_sdk/58_ask_oracle_tool/main.py
• uv run pytest tests/tools/ask_oracle/test_ask_oracle.py tests/sdk/conversation/test_switch_model.py -k 'ask_oracle or switch_profile_uses_custom_profile_store'
• uv run pytest tests/examples/test_examples.py --run-examples -k 58_ask_oracle_tool
• CI=true uv run python -m pytest -q tests/sdk

This PR was created by an AI agent (OpenHands) on behalf of the user.

@enyst can click here to continue refining the PR

---

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	eclipse-temurin:17-jdk	Link
python	amd64, arm64	nikolaik/python-nodejs:python3.13-nodejs22-slim	Link
golang	amd64, arm64	golang:1.21-bookworm	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:5a094a6-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-5a094a6-python \
  ghcr.io/openhands/agent-server:5a094a6-python

All tags pushed for this build

ghcr.io/openhands/agent-server:5a094a6-golang-amd64
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-golang-amd64
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-golang-amd64
ghcr.io/openhands/agent-server:5a094a6-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:5a094a6-golang-arm64
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-golang-arm64
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-golang-arm64
ghcr.io/openhands/agent-server:5a094a6-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:5a094a6-java-amd64
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-java-amd64
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-java-amd64
ghcr.io/openhands/agent-server:5a094a6-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:5a094a6-java-arm64
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-java-arm64
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-java-arm64
ghcr.io/openhands/agent-server:5a094a6-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:5a094a6-python-amd64
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-python-amd64
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-python-amd64
ghcr.io/openhands/agent-server:5a094a6-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-amd64
ghcr.io/openhands/agent-server:5a094a6-python-arm64
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-python-arm64
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-python-arm64
ghcr.io/openhands/agent-server:5a094a6-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-arm64
ghcr.io/openhands/agent-server:5a094a6-golang
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-golang
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-golang
ghcr.io/openhands/agent-server:5a094a6-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:5a094a6-java
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-java
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-java
ghcr.io/openhands/agent-server:5a094a6-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:5a094a6-python
ghcr.io/openhands/agent-server:5a094a6c50ef9af4ba1a476f7682df73c3dded8e-python
ghcr.io/openhands/agent-server:feat-ask-oracle-tool-python
ghcr.io/openhands/agent-server:5a094a6-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim

About Multi-Architecture Support

• Each variant tag (e.g., 5a094a6-python) is a multi-arch manifest supporting both amd64 and arm64
• Docker automatically pulls the correct architecture for your platform
• Individual architecture tags (e.g., 5a094a6-python-amd64) are also available if needed

[github-actions]

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

[github-actions]

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
openhands-sdk/openhands/sdk/conversation/impl
local_conversation.py	1076	88	92%	167, 341, 407, 659–660, 693, 735, 1063, 1087–1088, 1093, 1108, 1110, 1214, 1230, 1278, 1304, 1384, 1388–1394, 1459, 1479–1481, 1530, 1549–1551, 1863–1864, 1879, 2102, 2105–2106, 2131, 2162, 2168, 2249, 2256, 2259, 2262, 2266–2267, 2271–2272, 2275, 2282, 2307, 2311, 2314, 2333, 2385, 2388, 2427, 2434–2435, 2443, 2447–2449, 2456, 2493–2499, 2502, 2505, 2512, 2604, 2609, 2729–2730, 2748–2749, 2782, 2986, 2990, 3060, 3067–3068
openhands-tools/openhands/tools
__init__.py	14	2	86%	38–39
openhands-tools/openhands/tools/ask_oracle
definition.py	39	19	51%	47–53, 61–63, 65–69, 91–92, 95, 97
impl.py	31	21	32%	51–52, 57–58, 62–63, 70–71, 75–76, 81, 86, 90, 98–101, 109, 114–115, 120
TOTAL	41781	12046	71%

[all-hands-bot]

This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.

Summary

The ask_oracle tool is well-structured and consistently follows the existing auxiliary-LLM tool pattern established by vision_inspect and switch_llm: synchronous __call__, get_or_create_profile_llm for profile resolution, make_llm_completion for the one-shot call, and comprehensive error handling that returns actionable is_error observations rather than raising.

No material code issues found. The profile_store_dir parameter on LocalConversation is a clean, backward-compatible addition (defaults to None → ~/.openhands/profiles). Tests use real code paths (TestLLM, execute_tool) and cover the key edges: missing profile, empty response, no conversation, successful round-trip with message-shape verification. The OpenAPI allowlist entry follows the existing per-tool pattern.

Eval-risk flagging

This PR adds a new agent tool with behavior-guiding prompt language ("Treat the Oracle's response as strong guidance and follow its recommendation unless you have a clear reason not to"). While the tool is opt-in (not in BUILT_IN_TOOLS, requires an explicit Tool(name="ask_oracle") and a saved oracle profile), it still falls in the eval-risk category.

Noting that:

• Run tests ✅ passed for the current head SHA.
• All other CI checks (pre-commit, API breakage, REST breakage, Agent Server) ✅ passed.
• Run Integration Tests has not run for the current head SHA (194c8ce). The last integration test run (2026-06-11, 100% success) was against a substantially different version of the code (before the tool was moved from SDK builtins to openhands-tools).
• No eval monitor link or human eval confirmation is present.

Recommend a human maintainer decide after running lightweight evals / integration tests on the current head.

Risk Assessment

🟢 LOW — No bugs, security issues, or design flaws. The code is clean and consistent with established patterns. The only open item is eval validation for the current head SHA, which is a process gate, not a code defect.

[github-actions]

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

[github-actions]

🧪 Integration Tests Results

Overall Success Rate: 97.7%
Total Cost: $1.78
Models Tested: 5
Timestamp: 2026-08-26 22:29:48 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

• litellm_proxy_deepseek_deepseek_v4_flash: 📥 View & Download Logs
• litellm_proxy_anthropic_claude_sonnet_4_6: 📥 View & Download Logs
• litellm_proxy_gemini_3.1_pro_preview: 📥 View & Download Logs
• litellm_proxy_openai_gpt_5.5: 📥 View & Download Logs
• litellm_proxy_minimax_MiniMax_M2.7: 📥 View & Download Logs

📊 Summary

Model	Overall	Tests Passed	Skipped	Total	Cost	Tokens
litellm_proxy_deepseek_deepseek_v4_flash	100.0%	8/8	1	9	$0.02	459,609
litellm_proxy_anthropic_claude_sonnet_4_6	88.9%	8/9	0	9	$0.55	382,008
litellm_proxy_gemini_3.1_pro_preview	100.0%	9/9	0	9	$0.45	329,864
litellm_proxy_openai_gpt_5.5	100.0%	9/9	0	9	$0.76	282,147
litellm_proxy_minimax_MiniMax_M2.7	100.0%	8/8	1	9	$0.00	336,214

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

• Success Rate: 100.0% (8/8)
• Total Cost: $0.02
• Token Usage: prompt: 454,225, completion: 5,384, cache_read: 361,472, reasoning: 1,437
• Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_78092eb_deepseek_v4_flash_run_N9_20260826_222736
• Skipped Tests: 1

Skipped Tests:

• t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

litellm_proxy_anthropic_claude_sonnet_4_6

• Success Rate: 88.9% (8/9)
• Total Cost: $0.55
• Token Usage: prompt: 376,949, completion: 5,059, cache_read: 273,241, cache_write: 103,653, reasoning: 297
• Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_78092eb_claude_sonnet_4_6_run_N9_20260826_222738

Failed Tests:

• t02_add_bash_hello: Shell script is not executable (Cost: $0.06)

litellm_proxy_gemini_3.1_pro_preview

• Success Rate: 100.0% (9/9)
• Total Cost: $0.45
• Token Usage: prompt: 325,550, completion: 4,314, cache_read: 137,721, reasoning: 2,229
• Run Suffix: litellm_proxy_gemini_3.1_pro_preview_78092eb_gemini_3_1_pro_run_N9_20260826_222753

litellm_proxy_openai_gpt_5.5

• Success Rate: 100.0% (9/9)
• Total Cost: $0.76
• Token Usage: prompt: 277,044, completion: 5,103, cache_read: 172,544, reasoning: 1,818
• Run Suffix: litellm_proxy_openai_gpt_5.5_78092eb_gpt_5_5_run_N9_20260826_222737

litellm_proxy_minimax_MiniMax_M2.7

• Success Rate: 100.0% (8/8)
• Total Cost: $0.00
• Token Usage: prompt: 331,688, completion: 4,526, cache_read: 255,200
• Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_78092eb_minimax_m2_7_run_N9_20260826_222730
• Skipped Tests: 1

Skipped Tests:

• t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.