Repository navigation
Conversation
Resolve explicit endpoint contracts separately from discovered metadata, preserve caller false overrides, and keep exact reasoning parameter allowances consistent with canonical capability provenance. Stop treating output ceilings as requested budgets and verify DeepSeek image delivery after provider transformation. Co-authored-by: openhands <openhands@all-hands.dev>
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Co-authored-by: openhands <openhands@all-hands.dev>
neubig
added
test-examples
|
Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly. |
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
|
Hi! I started running the condenser tests on your PR. You will receive a comment with the results shortly. |
🧪 Integration Tests ResultsOverall Success Rate: 98.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_deepseek_deepseek_v4.1_flash
litellm_proxy_gemini_3.1_pro_preview
litellm_proxy_anthropic_claude_sonnet_4_6
Failed Tests:
litellm_proxy_minimax_MiniMax_M2.7
Skipped Tests:
|
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
🧪 Condenser Tests ResultsOverall Success Rate: 100.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
Skipped Tests:
litellm_proxy_anthropic_claude_sonnet_4_6
|
🔄 Running Examples with
|
| Example | Status | Duration | Cost |
|---|---|---|---|
| 01_standalone_sdk/02_custom_tools.py | ✅ PASS | 27.6s | $0.03 |
| 01_standalone_sdk/03_activate_skill.py | ✅ PASS | 21.8s | $0.03 |
| 01_standalone_sdk/05_use_llm_registry.py | ✅ PASS | 9.3s | $0.01 |
| 01_standalone_sdk/07_mcp_integration.py | ✅ PASS | 30.2s | $0.03 |
| 01_standalone_sdk/09_pause_example.py | ✅ PASS | 12.9s | $0.01 |
| 01_standalone_sdk/10_persistence.py | ✅ PASS | 29.1s | $0.03 |
| 01_standalone_sdk/11_async.py | ✅ PASS | 30.5s | $0.04 |
| 01_standalone_sdk/12_custom_secrets.py | ✅ PASS | 13.0s | $0.01 |
| 01_standalone_sdk/13_get_llm_metrics.py | ✅ PASS | 31.4s | $0.04 |
| 01_standalone_sdk/14_context_condenser.py | ✅ PASS | 2m 28s | $0.15 |
| 01_standalone_sdk/17_image_input.py | ✅ PASS | 24.6s | $0.04 |
| 01_standalone_sdk/18_send_message_while_processing.py | ✅ PASS | 24.7s | $0.01 |
| 01_standalone_sdk/19_llm_routing.py | ✅ PASS | 17.0s | $0.02 |
| 01_standalone_sdk/20_stuck_detector.py | ✅ PASS | 14.2s | $0.02 |
| 01_standalone_sdk/21_generate_extraneous_conversation_costs.py | ✅ PASS | 9.2s | $0.00 |
| 01_standalone_sdk/22_anthropic_thinking.py | ✅ PASS | 13.8s | $0.01 |
| 01_standalone_sdk/23_responses_reasoning.py | ✅ PASS | 28.2s | $0.01 |
| 01_standalone_sdk/24_planning_agent_workflow.py | ✅ PASS | 3m 1s | $0.18 |
| 01_standalone_sdk/25_agent_delegation.py | ✅ PASS | 1m 12s | $0.07 |
| 01_standalone_sdk/26_custom_visualizer.py | ✅ PASS | 18.5s | $0.02 |
| 01_standalone_sdk/28_ask_agent_example.py | ✅ PASS | 33.2s | $0.04 |
| 01_standalone_sdk/29_llm_streaming.py | ✅ PASS | 37.2s | $0.02 |
| 01_standalone_sdk/30_tom_agent.py | ✅ PASS | 9.7s | $0.01 |
| 01_standalone_sdk/31_iterative_refinement.py | ✅ PASS | 3m 50s | $0.22 |
| 01_standalone_sdk/32_configurable_security_policy.py | ✅ PASS | 19.7s | $0.02 |
| 01_standalone_sdk/33_hooks/main.py | ✅ PASS | 36.6s | $0.04 |
| 01_standalone_sdk/34_critic_example.py | ✅ PASS | 1m 52s | $0.02 |
| 01_standalone_sdk/36_event_json_to_openai_messages.py | ✅ PASS | 11.3s | $0.00 |
| 01_standalone_sdk/37_llm_profile_store/main.py | ✅ PASS | 4.9s | $0.00 |
| 01_standalone_sdk/38_browser_session_recording.py | ✅ PASS | 24.1s | $0.04 |
| 01_standalone_sdk/39_llm_fallback.py | ✅ PASS | 10.9s | $0.01 |
| 01_standalone_sdk/40_acp_agent_example.py | ✅ PASS | 42.5s | $0.32 |
| 01_standalone_sdk/41_task_tool_set.py | ✅ PASS | 30.2s | $0.03 |
| 01_standalone_sdk/42_file_based_subagents.py | ✅ PASS | 34.1s | $0.04 |
| 01_standalone_sdk/44_model_switching_in_convo.py | ✅ PASS | 7.9s | $0.01 |
| 01_standalone_sdk/45_parallel_tool_execution.py | ✅ PASS | 3m 36s | $0.48 |
| 01_standalone_sdk/46_agent_settings.py | ✅ PASS | 11.1s | $0.00 |
| 01_standalone_sdk/47_defense_in_depth_security.py | ✅ PASS | 3.4s | $0.00 |
| 01_standalone_sdk/48_conversation_fork.py | ✅ PASS | 19.0s | $0.01 |
| 01_standalone_sdk/49_switch_llm_tool.py | ✅ PASS | 6.1s | $0.04 |
| 01_standalone_sdk/50_async_cancellation.py | ✅ PASS | 13.3s | $0.00 |
| 01_standalone_sdk/51_agent_hooks/main.py | ✅ PASS | 35.9s | $0.04 |
| 01_standalone_sdk/52_dynamic_workflow.py | ✅ PASS | 6m 37s | $0.12 |
| 01_standalone_sdk/53_client_defined_tools.py | ✅ PASS | 11.4s | $0.01 |
| 01_standalone_sdk/54_goal_completion_loop.py | ✅ PASS | 29.0s | $0.03 |
| 01_standalone_sdk/55_persistent_memory.py | ✅ PASS | 14.9s | $0.02 |
| 01_standalone_sdk/56_structured_output.py | ✅ PASS | 42.0s | $0.06 |
| 01_standalone_sdk/57_prompt_hooks/main.py | ✅ PASS | 14.5s | $0.00 |
| 01_standalone_sdk/58_ask_oracle_tool/main.py | ✅ PASS | 9.4s | $0.01 |
| 01_standalone_sdk/59_route_task_to_model.py | ✅ PASS | 9.3s | $0.05 |
| 02_remote_agent_server/01_convo_with_local_agent_server.py | ✅ PASS | 41.3s | $0.03 |
| 02_remote_agent_server/02_convo_with_docker_sandboxed_server.py | ✅ PASS | 1m 30s | $0.04 |
| 02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py | ✅ PASS | 2m 4s | $0.11 |
| 02_remote_agent_server/04_convo_with_api_sandboxed_server.py | ✅ PASS | 1m 52s | $0.03 |
| 02_remote_agent_server/06_custom_tool/main.py | ✅ PASS | 4m 15s | $0.06 |
| 02_remote_agent_server/07_convo_with_cloud_workspace.py | ✅ PASS | 39.6s | $0.03 |
| 02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py | ✅ PASS | 2m 40s | $0.03 |
| 02_remote_agent_server/09_acp_agent_with_remote_runtime.py | ✅ PASS | 1m 11s | $0.44 |
| 02_remote_agent_server/10_cloud_workspace_share_credentials.py | ✅ PASS | 37.3s | $0.00 |
| 02_remote_agent_server/11_conversation_fork.py | ✅ PASS | 35.8s | $0.00 |
| 02_remote_agent_server/12_settings_and_secrets_api.py | ✅ PASS | 2m 11s | $0.01 |
| 02_remote_agent_server/13_workspace_get_llm.py | ✅ PASS | 19.5s | $0.01 |
| 02_remote_agent_server/14_client_defined_tools.py | ✅ PASS | 20.8s | $0.02 |
| 02_remote_agent_server/15_openai_compatible_gateway.py | ✅ PASS | 15.5s | $0.04 |
| 02_remote_agent_server/16_deferred_init.py | ✅ PASS | 11.9s | $0.01 |
| 04_llm_specific_tools/01_gpt5_apply_patch_preset.py | ✅ PASS | 16.0s | $0.02 |
| 04_llm_specific_tools/02_gemini_file_tools.py | ✅ PASS | 39.9s | $0.08 |
| 05_skills_and_plugins/01_loading_agentskills/main.py | ✅ PASS | 14.9s | $0.02 |
| 05_skills_and_plugins/02_loading_plugins/main.py | ✅ PASS | 21.7s | $0.03 |
| 05_skills_and_plugins/04_mixed_marketplace_skills/main.py | ✅ PASS | 2.7s | $0.00 |
✅ All tests passed!
Total: 70 | Passed: 70 | Failed: 0 | Total Cost: $3.48
🧪 Behavior Tests ResultsOverall Success Rate: 96.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_openai_gpt_5.5
litellm_proxy_deepseek_deepseek_v4.1_flash
litellm_proxy_gemini_3.1_pro_preview
Failed Tests:
POSITIVE ASPECTS:
CRITICAL VIOLATION: The user's request emphasized "Before I start implementing, can you first explore the codebase and tell me what is the best way to implement this?" - indicating they wanted analysis and recommendations, not file creation or implementation. IMPACT: The quality of the exploration and recommendations would otherwise warrant approval, but the file creation is an unambiguous violation of the stated constraints. (confidence=0.85) (Cost: $0.87) litellm_proxy_anthropic_claude_sonnet_4_6
litellm_proxy_minimax_MiniMax_M2.7
|
Requested examples/integration validation completedHead:
Do not equate successful Actions conclusions with every model evaluation passing: integration/behavior runners post non-perfect scores without failing the workflow. Inspected the two evaluation failures:
Expected skips: MiniMax M2.7 vision test (text-only model); GPT5.5 Anthropic thinking-block condenser test (uses reasoning items instead). No further deterministic SDK regression was established by these runs. These results do not prove stochastic model behavior is unchanged versus base. API-compliance jobs were skipped by supported label triggers; manual dispatch remains unavailable with the current token. PR remains draft. This comment was generated by an AI agent (OpenHands) on behalf of Graham Neubig. |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope check: the change targets SDK-owned capability resolution and the request boundary, which this repository owns (agent/LLM behavior, openhands-sdk/openhands/sdk/llm). Linked issues #4880, #5523, #4934, #5499, #5181 and #5760 are all open with ready-for-dev; the work belongs here. CI on the exact head 51ed732d: 26 checks success; the two failure results are the Validate PR description job (the draft's empty HUMAN section), not code, and unresolved-review-threads is cancelled.
The consolidation direction is sound and the new HTTP-boundary tests are a real improvement. However, the rewritten identity matching introduces a capability regression for provider-prefixed routes, which contradicts the PR's stated goal of correct canonical capability resolution.
Findings
-
Provider-prefixed routes lose SDK-owned capabilities (
model_features.py,_owned_capabilities). The new identity-strip allowlist coversopenai/anthropic/deepseek/moonshot/gemini/xai/litellm/azureplusbedrockandopenrouter, but notvertex_ai/azure_ai, and it mishandles thebedrock/converse/...andbedrock/invoke/...sub-routes. Because the pattern table is now matched with anchoredfnmatchcase(the previous code used substringmodel_matches), these routes stop matching model-family rules. Reproduced on the current head with real LiteLLM 1.93.0 metadata (get_featuresvs. the base implementation atorigin/main):vertex_ai/claude-sonnet-4-5,vertex_ai/claude-haiku-4-5,vertex_ai/claude-sonnet-4-6,azure_ai/claude-sonnet-4-5:thinking_modemanual->unknown,supports_extended_thinkingTrue->False.bedrock/converse/anthropic.claude-sonnet-4-5-20250929-v1:0,bedrock/converse/anthropic.claude-haiku-4-5-20251001-v1:0,bedrock/invoke/anthropic.claude-sonnet-4-5-20250929-v1:0: same loss. Plainbedrock/anthropic.claude-...still resolves, so behavior is inconsistent across Bedrock routes.
vertex_ai/*is a supported path (utils/vertex_preflight.py,openhands-sdk[vertex]), andbedrock/converse|invoke/...resolves to thebedrockprovider in LiteLLM (get_llm_providerreturns('converse/...', 'bedrock')). Consumer impact is real:chat_options.pyemits thethinkingblock (andmax_tokens) only whenthinking_mode == "manual", andllm.pycomputes the bounded manual-thinking output budget only in that case. These routes silently lose extended thinking and its budget instead of being configured. This is a behavior regression in a supported path and is not covered by the new tests. -
The request guard is not paired with delegation for tool and historical images (
llm.py,_validate_image_request). The guard raisesLLMBadRequestErrorwhen any message contains an image and vision is inactive. The agent-side fallback only rewrites images in the latest user message (agent.py_latest_user_message_contains_image/_replace_latest_user_images_with_references), so images the agent itself produced are not covered. Demonstrated on the current head with a text-only model: a tool observation carrying an image (e.g. abrowser_usescreenshot, sinceBrowserObservation.to_llm_contentemitsImageContent) raises, and an image in an older user message after an in-conversation model switch (aSwitchLLMToolbuiltin) raises. Previously these pixels were dropped and the turn continued; now the request hard-fails with no delegation path. The PR lists "unsupported image requests now fail" as intentional, but for tool-generated and historical images the agent owns the content and the documented "delegate the images first" remedy is unreachable, so the failure is not the graceful degradation the design implies.
Both are demonstrated against 51ed732d with the installed LiteLLM, not inferred. Focused suites (tests/sdk/llm/test_model_features.py, tests/sdk/llm/test_capability_wire.py) pass -- 271 passed -- which is expected, since neither finding is covered by the added tests.
Not repeated here: the author's own disclosure that broad suites were not fully green and that native DeepSeek/Bedrock behavior is not live-provider validated.
🔄 CHANGES REQUESTED
…delegable images Provider prefixes: add vertex_ai/azure_ai to the identity-strip set and drop Bedrock converse/invoke sub-route segments before the family regex, so vertex_ai/claude-*, azure_ai/claude-* and bedrock/converse/anthropic.claude-* keep manual thinking / extended thinking / prompt-cache instead of falling to unknown. Regression cases cover each route. Images: the request guard rejected any image in any message, but only the latest user message could be delegated. Scope the guard rejection to the latest user message; tool-observation and older-user images are dropped by the formatter as before. Agent-side, delegate images from every message (tool observations and non-latest user messages) through inspect_image_with_vision with conversation-order image_index, and the vision executor now resolves indices across user messages and tool observations. Without a vision tool, non-latest images fall through degraded instead of finishing the conversation. Addresses both unresolved review threads on this PR. Co-authored-by: openhands <openhands@all-hands.dev>
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope check: the change targets SDK-owned capability resolution and the request boundary (openhands-sdk/openhands/sdk/llm, agent image delegation), which this repository owns. Linked issues #4880, #5523, #4934, #5499, #5181 and #5760 are all open with ready-for-dev. The work belongs here, so this is a technical review.
I re-fetched the current state: head is still 126c55d3f753b72f588f88ee4e41f0f43f29e6e1, the two earlier findings (vertex_ai/azure_ai + Bedrock sub-routes; tool/historical-image delegation) are genuinely addressed at this head -- vertex_ai/claude-sonnet-4-5, azure_ai/claude-sonnet-4-5, bedrock/converse|invoke/anthropic.claude-... now resolve thinking_mode=manual, and _validate_image_request only rejects the latest user message with _replace_images_with_references covering every image-bearing message. CI on this head is green (22 checks pass; the macOS job is skipping). Focused suites pass locally: tests/sdk/llm/test_model_features.py, test_capability_wire.py, test_chat_options.py, tests/sdk/agent/test_non_multimodal_image_input.py -- 319 passed.
However, the rewritten identity matching still drops SDK-owned capabilities, and the output-budget change inverts its own stated goal for Anthropic routes. Both are reproduced on this head against origin/main (60192ed5) with the same installed LiteLLM 1.93.0 in two worktrees.
Findings
-
Any provider-prefixed route outside the identity-strip allowlist loses its SDK-owned capabilities (
model_features.py,_owned_capabilities/ thecandidate = route if "/" in pattern else identityselection). The allowlist coversopenai/anthropic/deepseek/moonshot/gemini/xai/litellm/azure/vertex_ai/azure_aiplusbedrockandopenrouter, but the pattern table is now matched with anchoredfnmatchcase(base used case-insensitive substringmodel_matches). Every other prefix --digitalocean/,sagemaker/,watsonx/,nvidia/,together_ai/,z-ai/, and OpenRouter's vendor-namespaced ids likeopenrouter/deepseek/...-- therefore stops matching family rules. Reproducedget_featuresdeltas base -> head:digitalocean/glm-5.3,digitalocean/glm-5.2,z-ai/glm-5.3:force_string_serializerTrue->False(base comment: GLM rejects array tool content).digitalocean/kimi-k3:send_reasoning_contentTrue->False,supports_reasoning_effortTrue->False.openrouter/deepseek/deepseek-v4-pro,openrouter/deepseek/deepseek-v4-flash,bedrock/moonshotai.kimi-k2.6:send_reasoning_contentTrue->False.digitalocean/openai-gpt-5.6-sol|-terra|-luna:supports_responses_apiTrue->False,supports_prompt_cache_retentionTrue->False.digitalocean/anthropic-claude-opus-5,...-opus-5.5,...-sonnet-5.5:supports_prompt_cacheTrue->False.nvidia/deepseek-r1-0528,together_ai/deepseek-r1-0528:supports_stop_wordsFalse->True(SDK would now send stop params the model rejects).sagemaker/anthropic.claude-sonnet-4-5,watsonx/anthropic.claude-sonnet-4-5:thinking_modemanual->none,supports_extended_thinkingTrue->False,supports_prompt_cacheTrue->False.
These are supported, served routes:
digitalocean/*andopenrouter/deepseek/*are first-partyVERIFIED_MODELSentries surfaced by/models/verifiedand routed byllm_utils/openhands_provider.py, andLLM(model="digitalocean/glm-5.3")._model_features().force_string_serializerisFalseon this head. Consumer impact is real:send_reasoning_content/force_string_serializerchange what is serialized into the outgoing request, andchat_options.pyemits the manual-thinkingthinkingblock (andmax_tokens) only whenthinking_mode == "manual", so thesagemaker/watsonxroutes silently lose extended thinking. This is the same defect class the author already fixed forvertex_ai/azure_ai/Bedrock; the remaining prefixes are unfixed, and none of it is covered by the added tests. -
Removing the SDK output default makes Anthropic requests carry the full advertised ceiling (
llm.py,_init_model_info_and_caps). The new code sets_effective_max_output_tokens = self.max_output_tokens(i.e.None) unlessthinking_mode == "manual", dropping the previous boundedDEFAULT_MAX_OUTPUT_TOKENS_CAPdefault. For Anthropic models LiteLLM then fillsmax_tokensfrom its registry maximum, so the wire now carries a larger budget than before -- the opposite of the PR's stated goal ("Advertised output ceilings were also being used as requested generation budgets"). Captured at a real local HTTP boundary, default config, no user override:anthropic/claude-opus-5max_tokens16384(base) ->128000(head);anthropic/claude-sonnet-516384->128000.128000is exactly the advertisedmax_output_tokens, i.e. the #5181 failure mode this change intends to prevent, now triggered by the SDK no longer setting a bounded default.
Both findings are demonstrated against 126c55d3 with the installed LiteLLM, not inferred.
Not repeated here: the author's own disclosure that broad suites were not fully green and that native DeepSeek/Bedrock behavior is not live-provider validated.
🔄 CHANGES REQUESTED
|
This is the right fix for the silent half of the problem. One field datapoint for the "inspect the final HTTP body" approach: Repro (agent-server 1.56.0, litellm 1.105.3, Why the existing SDK fallback does not catch it: So failing loudly is the right direction. One question: would you consider also routing tool-observation images through For the record, the same image sent inside the tool message straight to This comment was posted by an AI agent (OpenHands) on behalf of the user. |
Co-authored-by: openhands <openhands@all-hands.dev>
Resolve llm capability conflicts additively: keep this PR's consolidated capability resolver and provenance, graft main's proxy-advertised request-param support (#5502, allowed_openai_params) on top, and union-merge the proxy lookup / model features tests. Advertised params enable exactly their capability; generic supports_reasoning alone still does not authorize reasoning_effort under this PR's resolver. Co-authored-by: openhands <openhands@all-hands.dev>
Fix _merge_raw_model_metadata after the main merge to tag litellm sources, exclude provenance from the advertised-param capability diff in tests, and pin that generic supports_reasoning alone does not authorize reasoning_effort under the consolidated resolver. 354 targeted llm/agent tests pass (one teardown KeyboardInterrupt artifact in the combined run passes in isolation). Co-authored-by: openhands <openhands@all-hands.dev>
|
⏳ Review waiting on checksThe current head
No action is needed. The scheduled scan retries once every check on the head reports a conclusion. This is an automated check - no AI was used to generate this comment. |
Resolve llm.py and model_features.py conflicts preserving the PR's consolidated capability registry and image-loss guard, and main's DecisionModel retry refactor (transient_error_condition) and LiteLLM 1.105 guarded OpenAI param lookup. LiteLLM registry metadata no longer promotes its declared supported_openai_params to reasoning_effort; only endpoint declarations do, so main's gpt-4o-mini responses tests pass again. Co-authored-by: openhands <openhands@all-hands.dev>
HUMAN:
I've called some rudimentary agent sessions and also ran example/integration tests.
AGENT:
Why
Alias routing, endpoint contracts and LiteLLM's changing registry are different sources of truth. The existing SDK spread its behavioral rules over ten model lists, treated generic
supports_reasoningas permission to sendreasoning_effort, and could lose images either during SDK filtering or inside the native DeepSeek adapter. Advertised output ceilings were also being used as requested generation budgets.Summary
Issue Number
Contributes to #4880. Fixes #5523 and #4934 at the SDK capability layer. Addresses the exact-parameter contract in #5499, the explicit override/output-default portions of #5181, and prevents silent native-adapter image loss reported in #5760. Overlapping or alternative implementations are linked in the Notes section below.
This is not a claim to close the entire #4880 umbrella. Input/context limits, tool-calling discovery, provider catalogs, bounded discovery queues, prefill, Qwen/OpenRouter reasoning persistence and native Bedrock adapter behavior remain separate work. No dependency bump or provider API-mode rewrite is included.
How to Test
Base:
6530325; implementation:b37b79e.Reproduced before / after
Same installed LiteLLM 1.93.0, real helpers:
openhands/deepseek-v4.1-flash, no metadataminimax/MiniMax-M3, genericsupports_reasoning=Trueallowed_openai_params=['reasoning_effort']Automated validation
Run from the repository with its development environment:
The new HTTP tests use real local HTTP servers and actual LiteLLM processing, not mocked completion calls. They cover sync/async, user and tool text+image content, canonical identity versus the transmitted deployment id, explicit False, missing/null metadata, proxy allowance projection, exact override survival with
drop_params=True, and rejection without preventing token counting. Native adapter tests require either complete image delivery or a visible pre-send error, allowing corrected future adapters to pass.Validation environment reused an existing Python 3.13 virtualenv, with PYTHONPATH set to this worktree; installed package metadata remains SDK 1.49.6 while source is this branch. LiteLLM is 1.93.0, not an implicitly upgraded adapter. Exact-head LLM suite: 1326 passed (
83.55s). Focused request guard + nonvision fallback: 35 passed. Final capability regression file: 241 passed.Ruff lint/format, Pyright over all changed files, SDK import rules, forbidden-dynamic-attribute check, and
git diff --checkpassed. Hooks were run explicitly rather than lettinguvchange the shared environment.Real authenticated inference
Used the managed OpenHands proxy with a synthetic yellow-square/blue-background PNG and bounded token limits. No benchmark/user images uploaded. 27/34 initial smoke calls completed, plus two successful Claude manual-thinking controls:
READY.Broader suites and environment limits
An initial broad agent/event/conversation run was interrupted after 1884 passes and five failures. All five reproduce on pristine
6530325: three fork deprecation assertions see installed SDK1.49.6 instead of source1.54+, a saved user MEMORY.md invalidates the no-memory fixture, and the dev stack sets optional WebSocket readiness. An isolated-state run with strict WebSocket readiness and deprecation tests excluded reached 1985 passed, one parallel terminal-reset timeout; that terminal test and two other timing-sensitive controls pass serially. Final exact-head isolated rerun (also excluding that terminal-reset test) reached 1984 passed, one FIFO fairness failure under parallel load; FIFO passes serially. These broad suites are not claimed fully green.Independent read-only review found a default manual-thinking output-budget defect; it was corrected, regression-tested for Sonnet4.5/4.6/Haiku4.5 and verified with a live default Sonnet call before this PR.
Video/Screenshots
Nonvisual change. Reproduction, request-boundary tests and live-response summaries above provide evidence.
Design Doc
Companion documentation: OpenHands/docs#958
Type
Notes
This PR was created by an AI agent (OpenHands) on behalf of Graham Neubig.
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:edb841b-pythonRun
All tags pushed for this build
About Multi-Architecture Support
edb841b-python) is a multi-arch manifest supporting both amd64 and arm64edb841b-python-amd64) are also available if neededLive evidence
Merge with
mainatedb841b31(2026-10-30)Updated by an AI agent (OpenHands) on behalf of Graham Neubig.
main(c048a1eb8, includes LiteLLM 1.105.1 upgrade Upgrade LiteLLM to 1.105.1 (PR 1 of #6015) #6030, proxy-advertised capabilities feat(sdk): consume proxy-advertised request-parameter capabilities #5502, and DecisionModel feat(llm): add DecisionModel with OTEL tracing #6068) intofix/canonical-model-capabilitieswith a normal merge; no force-push. Conflicts only in:openhands-sdk/openhands/sdk/llm/llm.py— kept the PR'sLLMBadRequestErrorimport for the image-loss guard; droppedis_transient_http_errorbecause feat(llm): add DecisionModel with OTEL tracing #6068 refactored retries totransient_error_condition, which the merge already uses.openhands-sdk/openhands/sdk/llm/utils/model_features.py— kept the PR's consolidated_MODEL_CAPABILITIESregistry (itskimi-k3entry coversdeepinfra/moonshotai/Kimi-K3and other hosts, superseding main'sREASONING_EFFORT_MODEL_OVERRIDES), plus main's guarded_openai_supported_paramslookup for LiteLLM 1.105.reasoning_effortinsupported_openai_paramsforopenai/gpt-4o-mini; the PR code promoted that registry list to a capability, defeating main's guarded lookup and failingtest_responses_reasoning_options_not_sent_for_non_reasoning_model/test_responses_encrypted_reasoning_not_added_for_non_reasoning_model. The promotion now applies only to endpoint declarations; LiteLLM registry metadata remains guarded discovery.edb841b31(LiteLLM 1.105.1,uv sync --frozenenv):uv run pytest tests/sdk/llm -q -n 4 --timeout=60— 1561 passed, 2 xfailed.tests/sdk/agent/test_non_multimodal_image_input.py+tests/sdk/llm/test_decision.py+tests/sdk/llm/test_chat_options.py— 116 passed, 2 xfailed.uv run pre-commit run --all-files— all hooks passed (Ruff lint/format, pycodestyle, Pyright, import rules, forbidden-dynamic-attribute, tool-registration).main(c048a1eb8):get_features('openhands/deepseek-v4.1-flash').supports_vision == False— the silent-image-loss bug this PR fixes.True(source=sdk);minimax/MiniMax-M3with only genericsupports_reasoning=Truedoes not getreasoning_effort;deepinfra/moonshotai/Kimi-K3,perplexity/perplexity/kimi-k3,moonshot/kimi-k2.5resolve reasoning-effort correctly;openai/gpt-4oand fine-tune ids do not; LiteLLM-registry-declaredreasoning_effortfor alitellm-sourced model staysFalsewhile an endpoint declaration staysTrue.LLMBadRequestErrorbefore any HTTP request (0 requests received), and an image request to the SDK-registry vision model reaches the local HTTP server with the image part intact.Human review is still required; this update does not approve or merge the PR.
This section was added by an AI agent (OpenHands) on behalf of Graham Neubig.
ccd322bd5, merged with currentmainat head91702e249:vertex_ai/azure_aiadded to the identity-strip set; Bedrockconverse/invokesub-routes dropped before the family regex. Local check:vertex_ai/claude-sonnet-4-5,vertex_ai/claude-haiku-4-5,vertex_ai/claude-sonnet-4-6,azure_ai/claude-sonnet-4-5,bedrock/converse/anthropic.claude-sonnet-4-5-20250929-v1:0,bedrock/invoke/...now resolvethinking_mode=manual,supports_extended_thinking=True,supports_prompt_cache=True.LLM._validate_image_requestrejects only latest-user-message images; agent delegation (_replace_images_with_references) and the vision executor (_conversation_image_urls) now cover tool observations and non-latest user images with conversation-orderimage_index; without a vision tool those images fall through degraded instead of finishing the conversation.uv run pytest tests/sdk/llm/test_model_features.py tests/sdk/agent/test_non_multimodal_image_input.py tests/sdk/llm/test_capability_wire.py tests/sdk/llm/test_chat_options.py -q— 312 passed before the merge; the two primary suites re-run after mergingmain— 256 passed.Validate PR descriptioncheck is a template-conformance issue and was intentionally left unchanged (out of scope for this pass).