feat(tools): add per-call llm_profile override to the task tool - #4510
feat(tools): add per-call llm_profile override to the task tool#4510georgeglarson wants to merge 9 commits into
Conversation
|
📁 PR Artifacts Notice This PR contains a |
1735617 to
60752f7
Compare
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
3c64ac9 to
38a858e
Compare
Co-authored-by: openhands <openhands@all-hands.dev>
|
🚦 CI is currently failing on this PR's latest commit. Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request This is an automated check - no AI was used to generate this comment. |
|
🤖 OpenHands is reviewing this PR. Head commit: This comment was posted by an AI agent (OpenHands). |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.
Verdict
No material bugs or security issues found. The change is well-scoped and correctly implements the documented precedence (definition model → per-call llm_profile → parent model), reuses a single cipher-aware loading primitive (local_conversation.load_profile_llm) instead of duplicating store/decryption logic, preserves worker metrics and condenser isolation, and has solid test coverage. I ran the focused suite locally (tests/tools/task/test_task_manager.py -k "llm_profile or profile") plus the conversation/task tool-set tests — all passing — and CI at the head SHA is green across pre-commit, pyright, and the full test matrix.
Eval-risk flag (blocking approval)
This PR changes the task tool's description/prompt template (TASK_TOOL_DESCRIPTION) and adds new delegation behavior (llm_profile) — the "agent behavior that could plausibly affect benchmark/evaluation performance" category. There is no openhands-eval-monitor.vercel.app link and no human confirmation of benchmark results in the PR description or comments, so per repo policy I'm leaving this as COMMENT rather than APPROVE. A human maintainer should decide after running lightweight evals on the delegation path.
Risk assessment
🟡 MEDIUM — behavior-affecting tool/prompt change with no eval evidence; the implementation itself is low-risk, well-factored, and thoroughly tested.
HUMAN:
Enforced sub-agent delegation is the objective here.
AGENT:
Why
A parent agent cannot currently choose a different saved LLM profile for one delegated task. It must either switch its own model or use a subagent definition with a fixed
model:value.That makes mixed-model delegation awkward. The parent should be able to keep its model while assigning an individual task to a cheaper, faster, or specialized worker.
Summary
llm_profilefield to the task tool and advertise saved profile names so a parent can select a worker model per call.The selection order is:
llm_profile.A bare resume keeps the task's effective profile. Passing a different profile replaces it for that resume and later resumes. If the resumed subagent definition supplies its own model, the ignored per-call profile is cleared.
Unknown profiles return an error before task state is registered. There is no silent fallback to the parent model.
Issue Number
Fixes #4654.
How to Test
Result on the current branch:
All touched-file pre-commit hooks pass, including Ruff, Pyright, import dependency checks, and tool registration checks.
I also ran the delegation path against real saved profiles. The parent used MiniMax M3 and delegated a task through an encrypted MiMo v2.5 Pro profile:
The persisted profile did not contain the plaintext API key.
Video/Screenshots
Not applicable. This change affects SDK task-tool behavior and is covered by the live output above.
Design Doc
Per-call
llm_profiledesignCompanion documentation: OpenHands/docs#761
Type
Notes
The task tool description includes saved profile names from the default profile store so the parent model can select one. It does not include profile configuration, model IDs, provider URLs, or credentials. Profile names remain visible when
enable_switch_llm_tool=False; that setting removes the conversation-level switch tool but does not disable task-level worker selection.A file-defined subagent may use a custom
profile_store_dir. The task resolves profiles from that directory, but the advertised list still comes from the default store. Valid custom-store names may therefore be absent from the list.Once a task stores a per-call profile, resume can replace it with another profile but cannot explicitly return it to parent inheritance. A definition-owned model clears the stored override. Restoring task IDs across a new process remains outside the current in-memory
TaskManagercontract.Older SDK versions reject events containing the new field because action schemas forbid unknown fields. Mixed-version deployments should upgrade the agent server before emitting
llm_profile.Subscription-backed workers retain their factory-created condenser, matching current top-level agent creation and profile switching. Definition-level
model:loading still has its pre-existing encrypted-profile limitation; the new per-call path is cipher-aware.