docs(sdk): document ask_oracle tool

[enyst]

Summary

• Add an Ask Oracle guide under SDK → Agent Features.
• Document the required saved oracle profile, explicit tool enablement, stateless context boundary, combined usage metrics, and lack of fallback to the active model.
• Sync the runnable SDK example, which isolates the saved profile in a temporary directory rather than modifying the user's default profile store.

Validation

• npx --yes mintlify@latest broken-links (Node 22): no broken links
• SDK example harness: 1 passed, 69 deselected

Evidence

The .pr/ files were temporary, reviewer-only artifacts created so reviewers of the original SDK PR #3673 could inspect and verify the live behavior. By repository convention these artifacts are always deleted once their review purpose is complete; they were intentionally removed before merge and were never intended to ship. Their last historical snapshot is commit 78092eb9e:

• Validation summary
• Live result
• Reproduction script
• Focused test and example results

The live agent-loop validation was run with:

LLM_API_KEY=... LLM_BASE_URL=https://llm-proxy.eval.all-hands.dev ASK_ORACLE_PRIMARY_MODEL=openai/gpt-5.1 ASK_ORACLE_MODEL=openai/gpt-5-mini uv run python .pr/ask_oracle_live_validation.py

Its recorded output proves that the agent invoked the tool and received the Oracle response:

"agent_called_ask_oracle": true
"ask_oracle_observation_in_loop": true
"oracle_observation": "Short recommendation: each morning pick 1–3 clear, testable tasks ... ORACLE_LIVE_OK"
"final_agent_answer": "ORACLE_LIVE_OK — Focus each day on 1–3 testable coding tasks ..."
"passed": true

The checked-in example was also run through the repository harness:

LLM_API_KEY=... LLM_BASE_URL=https://llm-proxy.eval.all-hands.dev LLM_MODEL=openai/gpt-5.1 ASK_ORACLE_MODEL=openai/gpt-5-mini uv run pytest tests/examples/test_examples.py --run-examples -k 58_ask_oracle_tool

Recorded result:

EXAMPLE_COST: 0.01895175
1 passed, 69 deselected

Related SDK PRs: #3673, #4655

This PR was updated by an AI agent (OpenAI Codex) on behalf of the user.

[all-hands-bot]

🔍 Review in progress…

We are performing the review through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

Code Review

🟡 Acceptable — Solid guide overall, but two API/code-style inconsistencies with the established conventions in sibling guides should be resolved before merging.

The new agent-ask-oracle.mdx reads well, is correctly placed under SDK → Agent Features, and the frontmatter, Note/Warning blocks, RunExampleCode snippet, and cross-links to llm-profile-store, metrics, and custom-tools all follow the patterns set by the other agent guides. The explanation of the ask_oracle mental model ("consult another model, return control, never switch") is clearer than the equivalent prose in the LLM Profile Store guide. Two concrete issues below.

[IMPROVEMENT OPPORTUNITIES]

• [sdk/guides/agent-ask-oracle.mdx, Line 167] Inconsistency with sibling convention: conversation.state.stats.get_combined_metrics() is unique to this guide. Every other example in sdk/guides/*.mdx (agent-acp, agent-file-based, agent-settings, browser-session-recording, context-condenser, convo-custom-visualizer) reads cost via conversation.conversation_stats.get_combined_metrics(). If state.stats is genuinely the new canonical path, leave a one-line comment in the example explaining the divergence so readers don't trip over the inconsistency; otherwise align the example to conversation.conversation_stats to match the rest of the docs.

• [sdk/guides/agent-ask-oracle.mdx, Line 152] Inconsistency with the doc's own earlier example: The "Configure the Oracle Profile" section registers the tool as Tool(name=AskOracleTool.name) (line 59), but the ready-to-run example hard-codes Tool(name="ask_oracle") (line 152). Pick one and use it in both places. The constant form (AskOracleTool.name) is preferred — it removes a magic string and means a future rename of the tool is a one-line change in the SDK rather than a doc sweep.

• [sdk/guides/agent-ask-oracle.mdx, Line 134] Borderline comment: The inline comment # The Oracle model is saved under the conventional profile name "oracle". restates what ORACLE_PROFILE_NAME and usage_id="oracle" already convey. Either drop it or fold the explanation into the module-level docstring at the top of the file.

[STYLE NOTES]

None — the rest of the prose, the Note/Warning blocks, the use of the <RunExampleCode> snippet, and the navigation placement in docs.json are all consistent with the rest of the SDK guides.

[TESTING GAPS]

• [PR description] Missing evidence: This PR's main deliverable is a runnable example, and the only way readers can be confident the documented API (ask_oracle tool, ORACLE_PROFILE_NAME, profile lookup by name) actually works is to see it run end-to-end. Please add an Evidence section to the PR description with: (1) the exact command used (e.g. the uv run invocation from the example's docstring), and (2) the resulting output, including the EXAMPLE_COST line and at least one line showing the agent invoked ask_oracle and reported the Oracle's answer. Without this, reviewers and downstream users have to take the example on faith.

[RISK ASSESSMENT]

• [Overall PR] ⚠️ Risk Assessment: 🟢 LOW
  Documentation-only change. No executable surface in the docs repo; risk is purely "the example doesn't run as written," which the [TESTING GAPS] item above is designed to surface. No security, dependency, or breaking-change concerns.

VERDICT:
🟡 Worth merging after small fixes — Align the stats-access path with the sibling-guides convention (or document the divergence), pick one form of Tool(name=...) and use it in both places, drop the redundant inline comment, and add an Evidence section. None of these are blocking, but the stats inconsistency is the most likely to confuse readers copy-pasting between guides.

KEY INSIGHT: A doc PR that adds a new tool should establish — or consciously extend — the SDK's example conventions, not silently diverge from them. The conversation_stats vs state.stats and AskOracleTool.name vs "ask_oracle" mismatches are the kind of papercuts that compound across a docs set.

---

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

---

Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:

1. Add a .agents/skills/custom-codereview-guide.md file to your branch (or edit it if one already exists) with the /codereview trigger and the context the reviewer is missing (e.g., "Security concerns about X do not apply here because Y"). See the customization docs for the required frontmatter format.
2. Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.
3. When your PR is merged, the guideline file goes through normal code review by repository maintainers.

Resolve with AI? Install the iterate skill in your agent and run /iterate to automatically drive this PR through CI, review, and QA until it's merge-ready.

Was this review helpful? React with 👍 or 👎 to give feedback.

[enyst]

Addressed the AI review in a7601ad9:

• switched the example to conversation.conversation_stats
• registered the tool through AskOracleTool.name
• removed the redundant inline comment
• expanded the PR description's Evidence section with the exact validation commands, recorded EXAMPLE_COST, and output showing the agent's ask_oracle invocation and Oracle answer

The Evidence section links to SDK commit 78092eb9e, the final historical snapshot containing .pr/. Those files were temporary reviewer-only evidence for SDK PR #3673. They are always removed after serving that purpose, were intentionally deleted before merge, and were not intended to ship.

All three inline threads have replies and are resolved. The synchronized SDK-source follow-up is OpenHands/software-agent-sdk#4655.

This comment was created by an AI agent (OpenAI Codex) on behalf of the user.

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

🟡 Acceptable — Well-structured documentation addition with one minor line-focus annotation to fix.

This PR adds a clean, complete guide for the ask_oracle tool: a clear conceptual intro, a focused "How It Works" section, a correct comparison with switch_llm/switch_profile(), and a self-contained ready-to-run example. Navigation entry is correctly placed in the Agent Features group. All cross-references (llm-profile-store, metrics, custom-tools) resolve to existing pages.

One inline annotation is slightly off — see the comment below.

[RISK ASSESSMENT]

• [Overall PR] ⚠️ Risk Assessment: 🟢 LOW
  Pure documentation addition. No code, no dependencies, no behavioral changes. Safe to merge.

VERDICT:
✅ Worth merging: Core content is sound; one minor annotation nit to fix.

KEY INSIGHT:
The guide correctly draws the hard boundary between ask_oracle (stateless, single-shot consultation) and switch_profile()/switch_llm (persistent model switch) — exactly the distinction users need to understand.

---

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

---

Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:

1. Add a .agents/skills/custom-codereview-guide.md file to your branch (or edit it if one already exists) with the /codereview trigger and the context the reviewer is missing. See the customization docs for the required frontmatter format.
2. Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.
3. When your PR is merged, the guideline file goes through normal code review by repository maintainers.

Resolve with AI? Install the iterate skill in your agent and run /iterate to automatically drive this PR through CI, review, and QA until it's merge-ready.

Was this review helpful? React with 👍 or 👎 to give feedback.

[all-hands-bot]

🟡 Suggestion: focus={2, 5} highlights lines 2 and 5 of the code block, but line 5 is llm=primary_llm, — unrelated to AskOracleTool. The key line to emphasize is line 6 (tools=[Tool(name=AskOracleTool.name)],).

Change to focus={2, 6} so readers see the import (line 2) and the tools= argument (line 6) highlighted together, which is what this section is teaching.