Skip to content

Python: Send images in user messages to Bedrock Converse instead of dropping them - #8683

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 2 commits into
microsoft:mainfrom
kimnamu:fix/bedrock-gpt-6-images
Sep 24, 2026
Merged

Eduard van Valkenburg (eavanvalkenburg) merged 2 commits into
microsoft:mainfrom
kimnamu:fix/bedrock-gpt-6-images

Conversation

@kimnamu

@kimnamu Lucas Kim (kimnamu) commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Motivation & Context

Thanks for the Bedrock connector and for the Anthropic client's image mapping, which this PR mirrors. User-message images were skipped before Converse, so GPT-6 Sol/Luna/Astra answered blind.

Description & Review Guide

  • What are the major changes? A case "data" if content.has_top_level_media_type("image") branch returns a Converse image block via the existing _get_data_bytes (as Gemini does). Images outside user messages stay skipped. Tests cover png/jpeg (fail without the fix) and an assistant image.
  • What is the impact of these changes?
Red 64x64 PNG, "What color is this image? One word.", live us-east-1 Before After
us.openai.gpt-6-sol ❌ 'Unknown.', blocks sent ['text'] ✅ 'Red', blocks sent ['text', 'image']
us.openai.gpt-6-luna ❌ 'Gray', blocks sent ['text'] ✅ 'Red', blocks sent ['text', 'image']
us.openai.gpt-5.6-sol ❌ 'Unknown' ✅ 'Red'
us.openai.gpt-6-astra ❌ 'Unknown.', blocks sent ['text'] ✅ 'Red.', blocks sent ['text', 'image']
global.openai.gpt-6-astra ❌ 'Unknown', blocks sent ['text'] ✅ 'Red', blocks sent ['text', 'image']
Image in an assistant message (history) skipped, request OK ✅ unchanged (sending it gets a ValidationException from GPT-6 Sol/Luna/Astra)
Non-image data (application/octet-stream, text/plain) dropped ✅ still dropped, existing tests unchanged
Rich content in tool results dropped with a warning ✅ unchanged
Public API ✅ ✅ unchanged
$ python live.py   # before
us.openai.gpt-6-sol -> 'Unknown.' | user blocks sent ['text']
us.openai.gpt-6-luna -> 'Gray' | user blocks sent ['text']
$ python live.py   # after
us.openai.gpt-6-sol -> 'Red' | user blocks sent ['text', 'image']
us.openai.gpt-6-luna -> 'Red' | user blocks sent ['text', 'image']
$ python live.py us.openai.gpt-6-astra global.openai.gpt-6-astra   # before
us.openai.gpt-6-astra -> 'Unknown.' | user blocks sent ['text']
global.openai.gpt-6-astra -> 'Unknown' | user blocks sent ['text']
$ python live.py us.openai.gpt-6-astra global.openai.gpt-6-astra   # after
us.openai.gpt-6-astra -> 'Red.' | user blocks sent ['text', 'image']
global.openai.gpt-6-astra -> 'Red' | user blocks sent ['text', 'image']
  • What do you want reviewers to focus on? Format = lower-cased media subtype, with jpg mapped to jpeg. Subtypes Converse doesn't accept (its enum is [gif, jpeg, png, webp]), such as image/bmp or image/svg+xml, are skipped with a warning like other content this client can't send, so the text still goes through.

Related Issue

Fixes #8682

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

…ropping them

_convert_content_to_bedrock_block only handled text, function_call and
function_result, so image data content (Content.from_data with an image/*
media type) was skipped with a debug log and the model answered without
seeing the image.

Map image data content to a Converse image block with raw bytes, the
same way the Anthropic client maps image data to its image block.
Images in non-user messages and other data types are still skipped as
before.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings September 23, 2026 10:35
@agent-framework-automation agent-framework-automation Bot added the python Usage: [Issues, PRs], Target: Python label Sep 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Image formats require validation, and the private helper import needs its required Pyright suppression.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
What changed in this PR

Adds Bedrock Converse support for images in user messages while preserving existing behavior for assistant images.

Changes:

  • Converts PNG/JPEG user image data into Converse image blocks.
  • Adds regression coverage for user and assistant image handling.
File Description
python/​packages/​bedrock/​agent_framework_bedrock/​_chat_client.py Adds Bedrock image-block conversion and role filtering.
python/​packages/​bedrock/​tests/​test_bedrock_client.py Tests image conversion and assistant-image skipping.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Converse only accepts gif, jpeg, png and webp image blocks. Sending the
raw media subtype made image/jpg, upper-case types such as IMAGE/PNG, and
formats like image/bmp or image/svg+xml fail the whole request with a
ValidationException.

Lower-case the subtype and map jpg to jpeg. Other formats are skipped
with a warning, like other content this client cannot send, so the rest
of the message still goes through.

Co-Authored-By: Claude <noreply@anthropic.com>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lucas Kim (@kimnamu) The current head is green, and the dedicated full-diff review found no remaining actionable issues with Bedrock user-image forwarding, user-role filtering, supported-format validation, unsupported-format handling, or the accompanying regression coverage.

Merged via the queue into microsoft:main with commit e2c6df2 Sep 24, 2026
48 checks passed

This branch was previously deployed

1 inactive deployment
github-app-auth — 621ace90 Deployed Sep 24, 2026 by kimnamu via team_check #5204
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: BedrockChatClient drops images from user messages, so GPT-6 Sol/Luna/Astra answer without seeing them

3 participants