fix(llm): match LiteLLM proxy model_info by litellm_params.model too#3429
Draft
juanmichelini wants to merge 1 commit into
Draft
fix(llm): match LiteLLM proxy model_info by litellm_params.model too#3429juanmichelini wants to merge 1 commit into
juanmichelini wants to merge 1 commit into
Conversation
The proxy /v1/model/info matcher only checked `model_name` (the public alias). When a proxy config exposes a model under a short alias (`claude-opus-4-8`) for a provider id (`anthropic/claude-opus-4-8`), clients addressing the model by its provider id (the typical SDK setup of `litellm_proxy/anthropic/claude-opus-4-8`) missed the entry. When that happens, `get_litellm_model_info` falls back to `litellm.get_model_info` on the upstream catalog. For models LiteLLM already knows the fallback succeeded and hid the bug. For new model ids LiteLLM has not yet registered (Claude Opus 4.8 today), the fallback also returns None, `_model_info` ends up None, and `_supports_vision` returns False — so `model_info.supports_vision: true` overrides set on the proxy never reach the SDK and t08_image_file_viewing is skipped. Extend the matcher to also check `litellm_params.model`, mirroring how the proxy itself accepts incoming requests. The existing alias-match path is unchanged, so this is backward compatible. Adds dedicated unit tests for both match paths (alias and provider id), the negative case, and the end-to-end `get_litellm_model_info` path. Fixes #3428 Co-authored-by: openhands <openhands@all-hands.dev>
Contributor
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
Contributor
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Contributor
Coverage Report •
|
||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Even after All-Hands-AI/infra#1309 merged and deployed (which sets
model_info.supports_vision: trueon theclaude-opus-4-8proxy entry), the vision integration test is still skipped:See run #26608680950 on commit
45697f5.Root cause
openhands.sdk.llm.utils.model_info._get_model_info_from_litellm_proxyonly matched proxy entries bymodel_name:Our LiteLLM proxy config registers the new model with a short public alias:
The SDK is configured with
model = "litellm_proxy/anthropic/claude-opus-4-8". After stripping thelitellm_proxy/prefix it looks formodel_name == "anthropic/claude-opus-4-8"— a miss.get_litellm_model_infothen falls back tolitellm.get_model_info("claude-opus-4-8").claude-opus-4-7this fallback succeeds because LiteLLM upstream already knows the id, so the bug has been silently masked.claude-opus-4-8LiteLLM upstream does not know the id yet, the fallback also returnsNone,LLM._model_infoends upNone, andLLM._supports_vision()returnsFalse. The infra override is never read.This is a general bug, not specific to one model: any time the proxy uses an alias that differs from the provider id the SDK uses (the recommended LiteLLM pattern),
model_infooverrides set in proxy config are invisible to the SDK.What
Extend the matcher in
_get_model_info_from_litellm_proxyto also trylitellm_params.model— the same field the proxy itself accepts for incoming requests:The original
model_namematch path is unchanged, so callers that address the proxy by its alias still hit the same entry. The new path only fires whenmodel_namedoes not match — exactly today's silent-miss case.Tests
New
tests/sdk/llm/test_model_info_proxy_lookup.pycovers:test_lookup_matches_by_model_name_alias— backward-compat.test_lookup_matches_by_litellm_params_model— the case that brokeclaude-opus-4-8.test_lookup_returns_none_for_unknown_model— negative.test_get_litellm_model_info_uses_proxy_match_for_provider_prefixed_id— end-to-end throughget_litellm_model_info, assertingsupports_vision: trueflows through.Issue Number
Fixes #3428
How to Test
uv run pytest tests/sdk/llm/test_model_info_proxy_lookup.py -v uv run pytest tests/sdk/llm/ -k "model_info or vision or supports_vision" -q uv run pre-commit run --files \ openhands-sdk/openhands/sdk/llm/utils/model_info.py \ tests/sdk/llm/test_model_info_proxy_lookup.pyLocal results from this branch:
After merge, re-running the
Run Integration Tests vision claude-opus-4-8workflow onmainis expected to actually executet08_image_file_viewinginstead of skipping it.Type
Related
claude-opus-4-8on the proxymodel_info.supports_vision: true(correct on the proxy side, but invisible to the SDK without this fix)This PR was opened by an AI agent (OpenHands) on behalf of @juanmichelini.
@juanmichelini can click here to continue refining the PR
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:a5440ed-pythonRun
All tags pushed for this build
About Multi-Architecture Support
a5440ed-python) is a multi-arch manifest supporting both amd64 and arm64a5440ed-python-amd64) are also available if needed