Repository navigation
Python: AG-UI chat client drops multimodal message content #8114
Description
Description
The Python AGUIChatClient drops non-text content when serializing Agent Framework messages into AG-UI request messages.
In python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py, agent_framework_messages_to_agui currently handles text, function calls, and function results, but Content.from_uri() and Content.from_data() content is skipped. _client.py then sends the already-truncated message list to the HTTP service.
Reproduction
On the current main branch (26832bed1fa192febbb0ceec16d5a2900a39eb27), sending a normal user message through AGUIChatClient:
Message(
role="user",
message_id="user-1",
contents=[
Content.from_text("describe this"),
Content.from_uri("https://example.com/cat.png", media_type="image/png"),
Content.from_data(b"abc", media_type="image/png"),
],
)produces this outbound request message:
{"id": "user-1", "role": "user", "content": "describe this"}Both media items are absent. The same loss occurs through the public Agent -> AGUIChatClient -> HTTP path, before the request reaches the server. This was reproduced with an httpx.MockTransport, so no external model or service is involved.
Expected behavior
The client should preserve text and supported media content in their original order using AG-UI InputContent parts, for example:
{
"role": "user",
"content": [
{"type": "text", "text": "describe this"},
{
"type": "image",
"source": {
"type": "url",
"value": "https://example.com/cat.png",
"mimeType": "image/png"
}
},
{
"type": "image",
"source": {
"type": "data",
"value": "YWJj",
"mimeType": "image/png"
}
}
]
}AG-UI documents ordered multimodal input content for images, audio, video, and documents: https://docs.ag-ui.com/sdk/js/core/multimodal-inputs
Impact
Any Python Agent using AGUIChatClient to forward a multimodal user message to a stateless AG-UI server loses the media before the server can process it. This affects the regular client request path, not an exceptional or low-probability condition.
Proposed fix
- Convert Agent Framework URI and data content to AG-UI
InputContentparts. - Preserve the order of text and media parts.
- Keep the existing text-only and tool-call behavior backward compatible.
- Restrict InputContent lists to user messages because the AG-UI AssistantMessage content shape remains text-only.
- Add adapter and request-level regression tests covering URL, base64 data, and mixed text/media input.
This concerns the outbound Agent Framework-to-AG-UI conversion. It is distinct from #8083, which addresses multimodal content during orchestrator handoff.
Metadata
Metadata
Labels
Type
Projects
- Status
Activity
github-actions commentedon Sep 7, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
The bug occurs in python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py::agent_framework_messages_to_agui at lines 998-1059 when a Message contains Content.from_uri or Content.from_data alongside text. Minimal repro: convert one user Message containing text, an image URI, and image bytes, then observe that the result contains only the text string; _client.py passes this truncated value to post_run unchanged.
python/packages/ag-ui/tests/ag_ui/test_message_adapters.py::test_agent_framework_to_agui_preserves_multimodal_content1.2.2manjunathshiva commentedon Sep 7, 2026
I can reproduce this by inspection:
agent_framework_messages_to_agui(_message_adapters.py:1002-1017) branches ontext,function_callandfunction_resultonly, with no fallback, sourianddatacontents are dropped silently rather than raising or degrading. The outbound message is truncated before_client.pyposts it, as described.Worth noting for whoever picks this up: the pinned
ag-ui-protocol==0.1.19already exportsInputContent,TextInputContent,ImageInputContent,InputContentUrlSourceandInputContentDataSource, so the shape proposed here is achievable without touching the dependency floor.I'd be glad to take this, but I want to ask first rather than assume. The function this fix lands in also serializes
function_callandfunction_result, which puts the file on the AG-UI transport serialization path covered by the Contribution ownership clause indocs/specs/004-python-function-calling-loop.md— even though the change itself would only adduri/datahandling and wouldn't touch call/result behaviour. Is that inside or outside the clause for your purposes?Eduard van Valkenburg (@eavanvalkenburg), this is the same boundary question moonbox3 tagged you on in #8078, so a single answer would cover both.
vedantsonkar commentedon Sep 7, 2026
Hi Eduard van Valkenburg (@eavanvalkenburg), I’d like to contribute a fix for this issue as my first contribution here. Since it’s already assigned to you, is this available for me to work on?
My proposed approach is TDD: first add failing outbound adapter tests and a request-level regression test using httpx.MockTransport, then add serialization for ordered multimodal user content. I’ll preserve existing text-only and tool-call/result behavior and run the relevant regression, coverage, formatting, and typing checks.
I’m planning to keep this focused on outbound user-message serialization. Are there any ownership or function-calling-loop validation requirements you’d like me to account for before implementation?
vedantsonkar commentedon Sep 7, 2026
Following up after reviewing #8116: I’d like to propose an alternative that also preserves non-base64 data URIs and tests the public Agent.run() path in streaming and non-streaming modes.
In an isolated execution of #8116’s helper, data:text/plain,hello%20world becomes an inline data source containing hello%20world, although that source requires base64. Preserving the complete URI as a URL source would avoid this.
Eduard van Valkenburg (@eavanvalkenburg), would you welcome this alternative, and can you confirm the contribution ownership boundary and required validation scope for the shared serializer? The local environment is ready, all 128 existing adapter tests pass, and I’ve reproduced #8114 against unchanged code.
vedantsonkar commentedon Sep 7, 2026
I have created a draft PR with my understanding of the issue and the fix if this fits what you're looking for then great or else Im open to any changes you might suggest for the same.
#8117
CoralGarden52 commentedon Sep 7, 2026
Thanks Vedant Sonkar (@vedantsonkar) for pointing out the non-base64
data:URI case. I’ve added commit6515c9612to PR #8116 to preserve non-base64 data URIs as complete AG-UI URL sources, while keeping base64 data URIs as inline data sources, along with a regression test.Eduard van Valkenburg (@eavanvalkenburg), I’d appreciate it if you could review the updated PR when you have a chance. If you have any concerns or requested changes, I’m happy to revise it. I opened #8114 and am the original issue author; if appropriate, could you assign the issue to me?