Release v1.44.0 - #4684
Conversation
Co-authored-by: openhands <openhands@all-hands.dev>
|
Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly. |
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
🔒 Release Security Scan🔒 Approval drift (time-of-check vs time-of-use)❌ 5 finding(s) Baseline:
Audited 18 PR(s): 13 clean, 5 flagged, 0 un-auditable. 📦 Supply-chain dependency diff✅ no findings Baseline: OSV: no known vulns across 6 new/bumped dep(s). Bumped dependencies
Internal `openhands-*` bumps
Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed. |
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
🧪 Integration Tests ResultsOverall Success Rate: 97.7% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_deepseek_deepseek_v4_flash
Skipped Tests:
litellm_proxy_anthropic_claude_sonnet_4_6
Failed Tests:
litellm_proxy_gemini_3.1_pro_preview
litellm_proxy_openai_gpt_5.5
litellm_proxy_minimax_MiniMax_M2.7
Skipped Tests:
|
🔄 Running Examples with
|
| Example | Status | Duration | Cost |
|---|---|---|---|
| 01_standalone_sdk/02_custom_tools.py | ✅ PASS | 25.5s | $0.03 |
| 01_standalone_sdk/03_activate_skill.py | ✅ PASS | 21.9s | $0.03 |
| 01_standalone_sdk/05_use_llm_registry.py | ✅ PASS | 11.2s | $0.01 |
| 01_standalone_sdk/07_mcp_integration.py | ✅ PASS | 35.2s | $0.02 |
| 01_standalone_sdk/09_pause_example.py | ✅ PASS | 15.7s | $0.02 |
| 01_standalone_sdk/10_persistence.py | ✅ PASS | 33.3s | $0.03 |
| 01_standalone_sdk/11_async.py | ✅ PASS | 29.6s | $0.03 |
| 01_standalone_sdk/12_custom_secrets.py | ✅ PASS | 13.6s | $0.01 |
| 01_standalone_sdk/13_get_llm_metrics.py | ✅ PASS | 27.3s | $0.04 |
| 01_standalone_sdk/14_context_condenser.py | ✅ PASS | 2m 39s | $0.16 |
| 01_standalone_sdk/17_image_input.py | ✅ PASS | 20.9s | $0.02 |
| 01_standalone_sdk/18_send_message_while_processing.py | ✅ PASS | 22.8s | $0.02 |
| 01_standalone_sdk/19_llm_routing.py | ✅ PASS | 17.2s | $0.02 |
| 01_standalone_sdk/20_stuck_detector.py | ✅ PASS | 14.9s | $0.02 |
| 01_standalone_sdk/21_generate_extraneous_conversation_costs.py | ✅ PASS | 12.4s | $0.00 |
| 01_standalone_sdk/22_anthropic_thinking.py | ✅ PASS | 15.0s | $0.01 |
| 01_standalone_sdk/23_responses_reasoning.py | ✅ PASS | 59.6s | $0.01 |
| 01_standalone_sdk/24_planning_agent_workflow.py | ✅ PASS | 3m 43s | $0.30 |
| 01_standalone_sdk/25_agent_delegation.py | ✅ PASS | 1m 7s | $0.07 |
| 01_standalone_sdk/26_custom_visualizer.py | ✅ PASS | 18.9s | $0.02 |
| 01_standalone_sdk/28_ask_agent_example.py | ✅ PASS | 34.3s | $0.04 |
| 01_standalone_sdk/29_llm_streaming.py | ✅ PASS | 45.3s | $0.03 |
| 01_standalone_sdk/30_tom_agent.py | ✅ PASS | 12.9s | $0.02 |
| 01_standalone_sdk/31_iterative_refinement.py | ✅ PASS | 1m 34s | $0.09 |
| 01_standalone_sdk/32_configurable_security_policy.py | ✅ PASS | 23.1s | $0.02 |
| 01_standalone_sdk/33_hooks/main.py | ✅ PASS | 42.0s | $0.04 |
| 01_standalone_sdk/34_critic_example.py | ✅ PASS | 6m 23s | $0.66 |
| 01_standalone_sdk/36_event_json_to_openai_messages.py | ✅ PASS | 11.1s | $0.00 |
| 01_standalone_sdk/37_llm_profile_store/main.py | ✅ PASS | 11.9s | $0.00 |
| 01_standalone_sdk/38_browser_session_recording.py | ✅ PASS | 41.0s | $0.04 |
| 01_standalone_sdk/39_llm_fallback.py | ✅ PASS | 14.2s | $0.01 |
| 01_standalone_sdk/40_acp_agent_example.py | ✅ PASS | 46.3s | $0.33 |
| 01_standalone_sdk/41_task_tool_set.py | ✅ PASS | 37.1s | $0.04 |
| 01_standalone_sdk/42_file_based_subagents.py | ✅ PASS | 1m 3s | $0.06 |
| 01_standalone_sdk/44_model_switching_in_convo.py | ✅ PASS | 11.7s | $0.02 |
| 01_standalone_sdk/45_parallel_tool_execution.py | ✅ PASS | 3m 50s | $0.55 |
| 01_standalone_sdk/46_agent_settings.py | ✅ PASS | 12.1s | $0.01 |
| 01_standalone_sdk/47_defense_in_depth_security.py | ✅ PASS | 3.9s | $0.00 |
| 01_standalone_sdk/48_conversation_fork.py | ✅ PASS | 23.3s | $0.01 |
| 01_standalone_sdk/49_switch_llm_tool.py | ✅ PASS | 7.7s | $0.04 |
| 01_standalone_sdk/50_async_cancellation.py | ✅ PASS | 14.4s | $0.00 |
| 01_standalone_sdk/51_agent_hooks/main.py | ✅ PASS | 40.6s | $0.05 |
| 01_standalone_sdk/52_dynamic_workflow.py | ✅ PASS | 2m 20s | $0.10 |
| 01_standalone_sdk/53_client_defined_tools.py | ✅ PASS | 15.6s | $0.01 |
| 01_standalone_sdk/54_goal_completion_loop.py | ✅ PASS | 30.9s | $0.02 |
| 01_standalone_sdk/55_persistent_memory.py | ✅ PASS | 19.5s | $0.02 |
| 01_standalone_sdk/56_structured_output.py | ✅ PASS | 37.7s | $0.06 |
| 01_standalone_sdk/57_prompt_hooks/main.py | ✅ PASS | 14.6s | $0.00 |
| 01_standalone_sdk/58_ask_oracle_tool/main.py | ✅ PASS | 19.7s | $0.01 |
| 02_remote_agent_server/01_convo_with_local_agent_server.py | ✅ PASS | 38.9s | $0.02 |
| 02_remote_agent_server/02_convo_with_docker_sandboxed_server.py | ✅ PASS | 1m 42s | $0.04 |
| 02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py | ✅ PASS | 1m 7s | $0.06 |
| 02_remote_agent_server/04_convo_with_api_sandboxed_server.py | ✅ PASS | 1m 49s | $0.04 |
| 02_remote_agent_server/06_custom_tool/main.py | ✅ PASS | 5m 53s | $0.07 |
| 02_remote_agent_server/07_convo_with_cloud_workspace.py | ✅ PASS | 47.8s | $0.04 |
| 02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py | ✅ PASS | 4m 22s | $0.03 |
| 02_remote_agent_server/09_acp_agent_with_remote_runtime.py | ✅ PASS | 1m 36s | $0.42 |
| 02_remote_agent_server/10_cloud_workspace_share_credentials.py | ✅ PASS | 33.1s | $0.04 |
| 02_remote_agent_server/11_conversation_fork.py | ✅ PASS | 57.8s | $0.00 |
| 02_remote_agent_server/12_settings_and_secrets_api.py | ✅ PASS | 2m 16s | $0.02 |
| 02_remote_agent_server/13_workspace_get_llm.py | ✅ PASS | 29.8s | $0.01 |
| 02_remote_agent_server/14_client_defined_tools.py | ✅ PASS | 36.7s | $0.02 |
| 02_remote_agent_server/15_openai_compatible_gateway.py | ✅ PASS | 24.8s | $0.02 |
| 02_remote_agent_server/16_deferred_init.py | ✅ PASS | 23.1s | $0.01 |
| 04_llm_specific_tools/01_gpt5_apply_patch_preset.py | ✅ PASS | 34.6s | $0.03 |
| 04_llm_specific_tools/02_gemini_file_tools.py | ✅ PASS | 40.1s | $0.08 |
| 05_skills_and_plugins/01_loading_agentskills/main.py | ✅ PASS | 18.1s | $0.02 |
| 05_skills_and_plugins/02_loading_plugins/main.py | ✅ PASS | 17.7s | $0.02 |
| 05_skills_and_plugins/04_mixed_marketplace_skills/main.py | ✅ PASS | 12.4s | $0.00 |
✅ All tests passed!
Total: 69 | Passed: 69 | Failed: 0 | Total Cost: $4.13
🧪 Integration Tests ResultsOverall Success Rate: 96.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_deepseek_deepseek_v4_flash
litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_gemini_3.1_pro_preview
Failed Tests:
The agent successfully completed the primary objectives:
Violation of Evaluation CriteriaHowever, the agent violated a key evaluation criterion: "Verify that the agent did not over-verify the truncation limit change by running test suites much broader than necessary." Problematic actions:
Positive RecoveryThe agent did recover well:
AssessmentWhile the final deliverable was correct and the agent demonstrated good recovery from over-verification, the evaluation criteria explicitly warns against running "test suites much broader than necessary." The agent's attempt to run The agent should have stopped after the first successful test run on the targeted file and reported completion. (confidence=0.72) (Cost: $0.49) litellm_proxy_openai_gpt_5.5
litellm_proxy_minimax_MiniMax_M2.7
|
Release v1.44.0
This PR prepares the release for version 1.44.0.
Started by: @hieptl
Release Checklist
integration-test)behavior-test)test-examples)security-scan)release-note-requiredPRs are accurately called out in the final release notesWhat happens on merge
When this PR is merged, the
create-release.ymlworkflow will automatically:v1.44.0and auto-generated notes, plus an explicit preamble for mergedrelease-note-requiredPRspypi-release.ymlto publish all packages to PyPIversion-bump-prs.ymlto create downstream version bump PRs🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:bef8f59-pythonRun
All tags pushed for this build
About Multi-Architecture Support
bef8f59-python) is a multi-arch manifest supporting both amd64 and arm64bef8f59-python-amd64) are also available if needed