diff --git a/docs/decisions/0044-python-content-identity-and-protocol-metadata.md b/docs/decisions/0044-python-content-identity-and-protocol-metadata.md new file mode 100644 index 00000000000..129b50f91cb --- /dev/null +++ b/docs/decisions/0044-python-content-identity-and-protocol-metadata.md @@ -0,0 +1,379 @@ +--- +status: approved +contact: eavanvalkenburg +date: 2026-09-30 +deciders: eavanvalkenburg +consulted: +informed: +--- + +# Separate identity and protocol metadata in Python content + +## Context and Problem Statement + +Python `Content` and `Message` have first-class identity fields (`Content.id`, `call_id`, `status`, +`protected_data`, `name`, `file_id`, `server_name`, `Message.message_id`) and an open `additional_properties` +mapping. `to_dict()` persists `additional_properties` but drops `raw_representation`, so anything a provider or host +needs to replay persisted history must live in fields or `additional_properties`. + +There is no rule for which data goes where. On current `main`, providers and hosts have independently introduced +more than 25 metadata keys, and several cross package boundaries: + +| Problem | Examples on `main` | +| --- | --- | +| Provider item IDs have no fixed location | OpenAI Responses stores a function call's item ID in `fc_id`, hosted custom/tool-search items in `item_id`, and reasoning and computer-call item IDs in `Content.id`. Hosting Responses reads `fc_id` and emits it as its own output item ID. | +| Core behavior depends on provider keys | Core treats a function call as hosted whenever `additional_properties["server_label"]` is truthy. That one check controls approval handling, session resumption, skills and file-access auto-approval rules, and continuation error results. The same key is also persisted as the hosted server boundary in harness always-approve rules. Core streaming aggregation falls back to `additional_properties["item_id"]` to correlate code-interpreter fragments. | +| One concept, inconsistent representations | Function-call `status` is stored as metadata although `Content.status` exists. File names are stored as `filename` although `Content.name` exists. A hosted MCP server's name is `server_name` on MCP call content but `server_label` metadata on the call nested in a hosted approval. | +| Inconsistent key naming | `openai.responses.shell.output_type`, `openai.local_shell_command_parts`, `__foundry_reasoning_replay_item__`, `openai_content_type`, `computer_action_format`, and AG-UI's underscore-prefixed `_agui_collected_approval_occurrence`. | +| Stream merge semantics are implicit and inconsistent | Providers know whether an event is an incremental delta or a complete snapshot, but `Content` has no first-class way to preserve that distinction. Core therefore relies on type-specific implicit behavior instead of one documented contract. | +| Replay depends on non-persisted objects | OpenAI shell replay, Gemini tool-call parts, Hosting Responses output reuse, and a Foundry hosting OAuth fallback read `raw_representation`, which is absent after history is reloaded. | + +Each new provider feature adds another key, and consolidating the duplicated Responses conversions is not possible +while every adapter decides placement independently. + +[Spec 004](../specs/004-python-function-calling-loop.md) already defines function-call identity: +`function_call.Content.id` identifies one locally actionable Agent Framework occurrence, while `call_id` is the +provider's call/result correlation. [ADR 0039](0039-python-refusal-content.md) defines the flat +`model_output_kind="refusal"` marker. This decision keeps spec 004's identity semantics and the meaning of ADR +0039's marker unchanged, but moves that marker into the `core` namespace described below. It also changes which +field marks the hosted server boundary in spec 004's approval scenarios; the implementing change updates spec 004's +hosted-server-boundary scenario row and its authoritative test mapping. + +`additional_properties` merges are shallow and not uniform: text, reasoning, and function-call addition keeps the +existing value on collision, the optimized one-pass text and reasoning coalescer independently reproduces that +shallow merge, code-interpreter aggregation keeps the incoming value, and nested values are shared by copies. +Grouping metadata in nested mappings therefore requires changing how core merges and copies them. + +## Decision Drivers + +- Keep function-call occurrence identity unambiguous and separate from provider identifiers. +- Replay persisted history without `raw_representation`. +- Base core behavior, especially approval handling, on documented first-class fields rather than provider keys. +- Preserve a provider's explicit append-versus-replace intent instead of reconstructing it from payload values. +- Make each provider's metadata explicit and discoverable as a unit, separate from application data. +- Keep existing histories and pending approvals readable after upgrading. +- Keep independently versioned packages working together in one process. +- Keep core provider-neutral and avoid growing `Content` for provider-specific concepts. +- Behave predictably when streamed content is merged and copied. +- Give contributors a rule that is simple to follow and to test. + +## Considered Options + +1. Document the current keys without changing them. +2. Add first-class provider identity fields, such as `provider_item_id`, to `Content`. +3. Use flat prefixed keys in `additional_properties`, such as `openai.item_id`. +4. Group metadata in nested namespaces in `additional_properties`: one per provider, plus `core`. + +## Decision Outcome + +Chosen option: **"Group metadata in nested namespaces in `additional_properties`: one per provider, plus `core`"**, +combined with using existing first-class fields first. It gives each provider's data one explicit, typed home and +gives provider-neutral semantics a documented place instead of bare top-level names, without adding provider concepts +to core. In exchange, core must merge and copy namespaces one level deep. + +### Identity + +- `function_call.Content.id` is only the Agent Framework occurrence identity defined by spec 004. Adapters never + write a provider identifier to it. +- `function_result.Content.id`, when present, identifies the function-call occurrence that the result settles. It is + not a provider output-item ID. Host-owned results preserve the request occurrence ID; an id-less Host result is a + compatibility form that binds only when the active batch has one unambiguous matching `call_id`. +- `call_id` is the correlation between a call and its result. When a hosted protocol item exposes only one + identifier and Agent Framework needs to pair call and result contents, adapters use that identifier as `call_id`. + This replaces core's code-interpreter fallback on `item_id`. +- A provider item ID that has no first-class field is stored in the provider's namespace, for example + `additional_properties["openai"]["item_id"]` for OpenAI function calls and function results. +- Existing uses of `Content.id` for a provider identifier remain documented exceptions: reasoning items, computer + calls and results, and hosted-tool approval requests and responses, which both keep the provider's approval ID so + the provider can correlate the decision. Local approval requests and responses use the occurrence ID, as spec 004 + defines. New content types do not extend the exceptions without a decision. +- A host owns the identifiers it emits. It must not substitute one kind of identifier for another, such as a + `call_id` or occurrence ID as an output item ID. It reads upstream item IDs only from their documented location. + +### Hosted tool calls + +`function_call` content gets an explicit first-class `hosted` field. `True` means the provider or service executes +the call and Agent Framework must never execute it locally; `False` means a local call. Every current use of +`server_label` as a hosted marker switches to this field: in core's approval handling, session resumption, skills +and file-access auto-approval rules, and continuation error results, and in AG-UI, Foundry hosting, and Hosting +Responses. + +The hosted server's identity moves to the existing `server_name` field, which harness always-approve rules use as +the server boundary. Because routing no longer depends on it, `server_name` remains descriptive. Hosted approval +requests and responses continue to nest a `function_call`, so serializers keep their current shape. + +The exported `ToolApprovalRule` keeps its `server_label` attribute, constructor parameter, and serialized key +unchanged. Only the source of its value changes: new rules take it from the call's `server_name`. Stored standing +rules therefore need no migration, and matching compares a rule's `server_label` with the call's `server_name`. + +Adapters set `hosted=True` on every call the provider executes, including the call nested in a hosted approval +request. A provider-issued approval left unmarked would be treated as a local call, so every adapter that parses +provider approvals must test that it sets the field. The field records who executes the call; it does not establish +trust. Trust still comes from where the content originated: spec 004's trusted pending snapshot for local approvals, +and the provider for hosted ones. + +Two alternatives were rejected. Nesting `mcp_server_tool_call` in hosted approvals would still infer routing from +content type, and would change every approval serializer. Setting `server_name` on the nested call would turn a +descriptive field into a routing signal that a local tool adapter could reasonably set later. + +#### Legacy records and older producers + +- `Content.to_dict()` always writes `hosted` for `function_call` content, as `true` or `false`. A record that contains + `hosted` is a new record: its value must be a boolean, anything else is rejected, and it always takes precedence + over `server_label`. +- When `hosted` is not supplied, the constructor infers it: `True` if `additional_properties["server_label"]` is set, + otherwise `False`. Inferring `True` emits a `FutureWarning` identifying the legacy `server_label` form. This covers + records written before this change and older provider packages that construct content on a newer core. +- The server boundary is migrated the same way. When a function call has no `server_name`, the constructor takes it + from `additional_properties["server_label"]`. When both are present they must be equal; a mismatch is rejected. + No producer sets `server_name` on function calls today, and new writers dual-write the same value, so a mismatch + indicates a malformed record rather than one that should be reconciled. +- `Content.from_dict()` builds content through the constructor, so the same inference applies to messages, + responses, sessions, history providers, harness approval state, and calls nested in approval requests. +- Workflow checkpoints pickle `Content` and restore it through `__setstate__`, which bypasses `from_dict()`. + `__setstate__` applies the same inference and warning. +- Hosts, AG-UI, and DevUI build content from their own wire formats; they set `hosted` explicitly rather than rely on + inference. + +#### Streaming aggregation + +Merging streamed function-call parts rebuilds the content from a fixed list of fields, so the list must include +every first-class field this decision relies on: + +- `hosted`: the merged call is hosted if either part is hosted. Parts that do not carry the flag default to local, + so rejecting differing values would break streams whose argument deltas omit it, and the unsafe error is treating + a hosted call as local. +- `server_name`: preserved; differing non-empty values are rejected, as for `call_id`. +- `status` on function calls and reasoning: preserved; the latest non-empty value wins, because status progresses + during a stream, for example from `in_progress` to `completed`. + +All content types that core can aggregate get an explicit first-class +`merge_mode: Literal["append", "replace"] | None` field. This initially covers `text`, `text_reasoning`, +`function_call`, `usage`, `code_interpreter_tool_call`, and `code_interpreter_tool_result`. + +- The mode belongs to the incoming stream fragment. It is not provider metadata and does not participate in + identifying which logical item the fragment belongs to. Core first applies the existing type-specific identity + and adjacency rules, then applies the fragment's mode. +- Direct `Content.__add__` and optimized response coalescers honor the same modes. Constructors reject a non-`None` + mode on content types that core does not aggregate. +- `append` uses the content type's existing incremental merge. `replace` treats the incoming semantic payload as a + complete snapshot and replaces the accumulated payload for that logical item. Stable identity and correlation + fields are still validated, while status and namespaced metadata follow their separate merge rules. +- Providers set a mode on every mergeable streamed item they produce. Delta events use `append`; authoritative + payload snapshots use `replace`. A `*.done` event that carries only final metadata and omits earlier streamed + payload remains incremental. Core does not use prefix comparison or another payload heuristic when a mode is + present. +- `None` means existing type-specific behavior, not `append`. This keeps older provider packages working on a newer + core. Removing an implicit fallback is a separate compatibility decision after supported providers mark their + fragments explicitly. +- The field is serialized so individually persisted updates retain their merge semantics. Aggregation consumes it; + finalized `ChatResponse` and `AgentResponse` content has `merge_mode=None`, so an operational stream instruction + does not become part of persisted logical history. +- This is one field interpreted by the existing type-specific merge code. It does not introduce merge strategy + objects, a registry, or provider callbacks. + +`ChatResponseUpdate` and `AgentResponseUpdate` carry the same optional field for whole-message snapshots. An incoming +update with `merge_mode="replace"` clears the accumulated contents of the logical message selected by `message_id` +and role before applying the incoming contents. This is required for protocol snapshots that can replace a +heterogeneous message or artifact, because URI and binary content are not individually mergeable. `append` and +`None` retain the existing message-content accumulation behavior. + +### Metadata placement + +```python +content.additional_properties == { + "core": {"model_output_kind": "refusal"}, # provider-neutral, documented by core + "openai": {"item_id": "fc_123"}, # owned by agent-framework-openai + "foundry": {"reasoning_item": {...}}, # owned by agent-framework-foundry + "_approval_request_id": "...", # core-private, unchanged + "ticket": "ABC-42", # application data, unchanged +} +``` + +1. **Use first-class fields first.** Existing fields such as `status`, `protected_data`, `name`, `file_id`, and + `server_name` are used where they apply. Missing constructor parameters for existing fields are added, such as + `status` on function calls, function results, and reasoning, `id` on function results, and `name` on data and URI + content. No new fields are added for provider-specific concepts; the new `hosted` and `merge_mode` fields record + framework behavior. +2. **Provider data lives in the provider's namespace.** `additional_properties[""]` is a mapping owned by + one integration package and named after it: `openai`, `foundry`, `anthropic`, `gemini`, `bedrock`, `a2a`, + `ag_ui`, and so on. Data for a wire protocol belongs to the package that owns that protocol's mapping: Foundry chat + and the hosts that implement Responses read and write `openai` for Responses data, and Foundry keeps Foundry-only + data under `foundry`. Each owner documents its keys in its package README and may publish a `TypedDict` for its + namespace and small accessor functions. +3. **Provider-neutral data lives in `core`.** `additional_properties["core"]` holds semantics documented by + core that any package may read or write: `model_output_kind` from ADR 0039 and the caller-facing + `prompt_cache_breakpoint`. Only core adds keys to `core`. File names use the first-class `Content.name` field. +4. **Namespaces are exactly one level deep.** A namespace maps flat key names to plain JSON values (mappings, lists, + strings, numbers, booleans, and `None`), never dataclass or Pydantic instances, because `Content.from_dict()` does + not reconstruct arbitrary types. An owner that implements several protocols qualifies key names where needed rather + than adding another level. Values inside a namespace are opaque to merging and treated as immutable. +5. **Other top-level keys.** Leading-underscore keys remain core-private at the top level. Keys defined by other + decisions, such as the security labels of ADR 0024, are not moved by this decision. Markers that exist only while + building a request are never stored on Agent Framework objects. All other top-level keys belong to applications; + `core` and the namespace names are reserved. +6. **Persisted replay data is defined per key.** Data that must survive persistence uses `protected_data` for opaque + provider payloads or a namespace key whose owner documents its exact shape. The documentation states whether the + value can contain encrypted or protected data, what is sanitized or excluded, and that only adapters write it. + Adapters store only fields required for replay, not converted SDK objects, and reject malformed values when + replaying. `raw_representation` may be used as a fast path only when the persisted path gives the same result or + fails explicitly. + +#### Merging and copying namespaces + +With today's shallow merges, a namespace would be replaced as a whole, losing fields from earlier stream fragments or +from another producer. Core therefore changes how `additional_properties` are combined and copied: + +- One core merge function combines two mappings. When both hold a mapping under the same top-level key, it merges + them one level deep. Each merge site keeps its existing conflict rule for colliding keys: content addition keeps + the existing value; code-interpreter and response aggregation keep the incoming value. +- Every core merge site uses it: direct content addition, one-pass text and reasoning coalescing, code-interpreter + aggregation, response aggregation, workflow-agent response merging, and function-result carriers. Packages that + merge metadata themselves, such as A2A combining task metadata, use it too. +- The rule applies to any top-level mapping, not only known namespace names, because core cannot list every + third-party provider. Application values that are mappings therefore also merge one level deep when streamed + fragments are combined, instead of being replaced. +- `Content` and `Message` constructors copy namespace mappings one level, so constructed objects don't share them. + Writers never change a namespace in place; they replace it with an updated copy. Core provides small public helpers + to read a namespaced value and to store an updated copy, so providers outside this repository follow the same rule. +- Compaction's token estimation strips opaque values inside namespaces as well as at the top level. Today it removes + only a top-level `encrypted_content`, so the encrypted payload inside Foundry's persisted reasoning item is counted. + +Core documents `core`, the namespace rules, and the merge behavior in its package documentation. There is no +central allowlist and no runtime validation; `additional_properties` remains open to applications outside the +reserved names. + +### Compatibility + +- Readers accept legacy locations indefinitely. Removing a legacy read is a separate breaking-change decision. +- **Upgrading is safe; downgrading is not.** New runtimes read every existing history: content written before this + change, with or without `server_label`, loads through the legacy inference and legacy key reads described above. + The break is in the other direction: `hosted` and `merge_mode` are new serialized fields, and `Content`'s + constructor rejects fields it does not know, so runtimes that predate this change fail to load records a new + runtime wrote while either field is present. Aggregated logical history clears `merge_mode`, but persisted + individual updates can contain it. Downgrades and mixed-version deployments that share records across this + boundary are not supported. The implementing change is marked breaking for that reason, and its release notes say + that existing histories remain readable after upgrading. +- For replay-critical keys, writers write both the new and legacy locations until the writing package's next major + version. This keeps independently versioned packages in one process working, for example an older Hosting + Responses reading `fc_id`, or older AG-UI, Foundry hosting, and Hosting Responses reading `server_label` from + content that a newer provider package produced. +- `model_output_kind` and `prompt_cache_breakpoint` move into `core`; their meaning is unchanged. Readers accept the + flat keys indefinitely, writers write both until their next major version, and applications that set the flat keys + keep working. ADR 0039 remains the record of the refusal decision. +- A package that writes namespaces requires, as its minimum core version, the core release that merges them. On an + older core, streamed fragments would lose namespace fields during aggregation. +- Keys used only for display, diagnostics, or a single request switch directly. + +Response-level `additional_properties` are out of scope. `ChatResponse.to_dict()` does not persist them by default, +so preserving response-level metadata would need its own serialization decision. New response-level keys should +nevertheless use the same namespaces; response aggregation uses the namespace-aware merge, so they combine safely. + +### Consequences + +- Good, because each identifier has one meaning, and function-call occurrence identity cannot be confused with + provider identity. +- Good, because approval handling and auto-approval rules depend on an explicit field rather than on the presence + of a provider string. +- Good, because each provider's metadata sits under one explicit key that can be typed as one `TypedDict`, inspected, + or stripped as a unit, and is visibly separate from application data. +- Good, because provider-neutral semantics have one documented home instead of competing with application keys. +- Good, because aggregation uses provider-declared delta or snapshot semantics instead of guessing from payload + prefixes. +- Good, because shared Responses conversion can later rely on one documented set of locations. +- Good, because persisted histories can be replayed after reload, and what they store is explicit. +- Bad, because core's merge behavior changes at every merge site, including for application values that are mappings. +- Bad, because writers must replace namespaces rather than mutate them; a writer that mutates in place corrupts + metadata shared with copies. The core helpers make the safe path easy but cannot enforce it. +- Bad, because core gains public helper functions, and writer packages must raise their minimum core version. +- Bad, because ADR 0039's marker and the caller-facing `prompt_cache_breakpoint` move, so samples and documentation + change, although the flat keys remain readable. +- Bad, because `core` and every namespace name become reserved top-level keys. No collisions exist in this + repository, but applications outside it could already use those names. +- Bad, because writers carry duplicated legacy keys until their next major version. +- Bad, because runtimes that predate either new field cannot read records that still contain that field, although + upgraded runtimes read all existing records. +- Bad, because `Content` gains an operational field that is meaningful only before aggregation; clearing it from + finalized content keeps that concern from leaking further. +- Bad, because every provider that emits mergeable stream content must classify its events, and a wrong `replace` + marker discards accumulated payload while a wrong `append` marker can duplicate a snapshot. +- Bad, because unmarked fragments retain their existing type-specific behavior during compatibility, including any + ambiguity that the explicit mode is intended to remove. +- Bad, because a provider adapter that forgets to set `hosted` makes a hosted call look local; only adapter tests + catch this. +- Neutral, because existing `Content.id` exceptions remain, although they are now documented and closed. + +### Validation + +- Each package's tests assert that converting its fixture corpus writes only keys in its own namespace, documented + `core` keys, core-private keys it intentionally sets, and legacy keys it still dual-writes. +- Core tests cover namespace merging at each merge site with both conflict rules, one-level depth, non-mapping values + under a namespace name, constructor copying, the helpers' copy-on-write behavior, and compaction stripping opaque + values inside namespaces. +- OpenAI and Foundry streaming tests merge fragments whose namespaces carry different keys and assert that none is + lost. +- Every replay-critical path has persisted-history fixtures with legacy-only, dual-written, and new-only metadata. +- Replay tests serialize and reload history, so `raw_representation` is absent. +- Core tests cover `hosted` round trips, rejection of non-boolean values, precedence over `server_label`, and legacy + inference with its warning through the constructor, `from_dict()`, nested approval calls, and pickling. They also + cover filling `server_name` from `server_label` and rejecting records where the two differ. +- Streaming aggregation tests merge function-call and reasoning parts and assert that `hosted`, `server_name`, and + `status` survive, following the rules above. +- For every mergeable content type, contract tests split a known payload into different append fragments, optionally + follow them with a replace snapshot, and assert exact reconstruction through both `ChatResponse.from_updates()` and + streaming `get_final_response()`. Tests include repetitive and prefix-shaped fragments, serialization of + unaggregated updates, rejection of invalid modes and unsupported content types, clearing the mode on finalized + content, and unmarked compatibility behavior. +- Harness tests load standing `ToolApprovalRule` state written before the change and confirm it still matches hosted + calls from the same server, and never matches a local call with the same name and arguments. +- Every adapter that parses provider-executed calls or approvals asserts that it sets `hosted=True`. Approval tests in + core, OpenAI, Foundry hosting, Hosting Responses, and AG-UI cover new and legacy records, including approval-response + round trips that keep the provider approval ID. + +## Pros and Cons of the Options + +### Document the current keys without changing them + +- Good, because nothing changes for users. +- Bad, because approval handling still depends on `server_label`, and replay still depends on raw objects. +- Bad, because nothing prevents the next ad hoc key. + +### Add first-class provider identity fields + +- Good, because provider item IDs become explicit and typed. +- Bad, because provider concepts enter the provider-neutral core API. +- Bad, because every other provider-specific value would still need a separate rule. + +### Use flat prefixed keys + +- Good, because existing shallow merges keep working unchanged and no core helpers are needed. +- Good, because it extends the existing `openai.responses.*` shell keys. +- Bad, because a provider's data is spread across many top-level keys mixed with application data, and cannot be + typed, inspected, or stripped as a unit. +- Bad, because provider-neutral keys would stay as bare top-level names that compete with application keys. + +### Group metadata in nested namespaces, one per provider plus `core` + +- Good, because each provider's metadata is explicit, typed, and separable from application data. +- Good, because provider-neutral semantics have a documented home. +- Bad, because core merges must become namespace-aware, writers must copy on write, and writer packages need a newer + core. +- Bad, because it requires migrating existing keys with a compatibility period. + +## More Information + +### Implementation specification + +[Spec 005](../specs/005-python-content-identity-and-protocol-metadata.md) contains the exact source-to-target mapping +for every field and metadata key, its owning package, serialized shape, compatibility rule, and authoritative tests. +This ADR records the placement and identity rules, not an inventory of implementation details. Unreleased intermediate +forms are removed during implementation rather than documented as compatibility formats. + +### Related documents + +- [Spec 004: Python function-calling loop](../specs/004-python-function-calling-loop.md) +- [Spec 005: Python content identity and protocol metadata](../specs/005-python-content-identity-and-protocol-metadata.md) +- [ADR 0039: Preserve model refusals with marked Python text content](0039-python-refusal-content.md) +- [ADR 0043: Preserve host runtime context through MCP invocation and approval](0043-python-mcp-runtime-context.md) +- [Issue 8910: Explicit append/replace semantics for streamed content merging](https://github.com/microsoft/agent-framework/issues/8910) +- [Issue 8903: Streamed code-interpreter code can be corrupted by prefix inference](https://github.com/microsoft/agent-framework/issues/8903) diff --git a/docs/specs/004-python-function-calling-loop.md b/docs/specs/004-python-function-calling-loop.md index 440660d6a71..d5b9d1ec622 100644 --- a/docs/specs/004-python-function-calling-loop.md +++ b/docs/specs/004-python-function-calling-loop.md @@ -263,6 +263,19 @@ the tool-call group, and provider adapters serialize or reconstruct the provider that occurrence id once and preserve it through streaming aggregation, serialization, and replay; existing ids are never regenerated. Deserializing a legacy stored call without `Content.id` does not manufacture a new identity. +`Content.id` on a `function_result`, when present, identifies the occurrence it settles. An exact result occurrence +identity takes precedence over `call_id`; an id-less Host result is a compatibility form that binds only when the +active batch has one unambiguous unanswered occurrence with that `call_id`. + +`Message.message_id` identifies the containing provider/protocol message and groups its streamed updates. It is not a +function-call occurrence ID or call/result correlation ID. The message-level identity and metadata contract is +defined by [spec 005](005-python-content-identity-and-protocol-metadata.md). + +`function_call.hosted` decides execution ownership. A hosted call is provider-executed and must never be executed +locally. `function_call.server_name` is the descriptive server boundary used by standing approval rules; it does not +decide execution ownership. Legacy `additional_properties["server_label"]` records are migrated as defined by +[spec 005](005-python-content-identity-and-protocol-metadata.md). + New local approval requests use the function call occurrence id as their request id. Provider-issued hosted approval request ids remain unchanged and continue to follow the hosted service protocol. A response carrying the occurrence id can bind without embedding a function call because the trusted pending snapshot is authoritative. For an @@ -682,7 +695,7 @@ that manually replay messages own the equivalent rule: do not resend an approval | Shared call budget | Auto-approved re-entry does not reset `max_function_calls`, and every executed approval group counts even when it pauses for input. | `test_tool_approval_middleware_auto_approved_loops_share_function_call_budget`, `test_approval_resume_user_input_counts_toward_function_call_budget` | | Standing tool rule | Tool-level approval applies only to later matching tools. | `test_tool_approval_middleware_always_approve_tool_rule` | | Forged standing rule | An unbound or substituted hosted response cannot create a standing middleware approval rule for caller-selected metadata. | `test_tool_approval_middleware_drops_forged_standing_approval`, `test_tool_approval_middleware_rebinds_hosted_standing_approval` | -| Hosted server boundary | Standing approval does not cross `server_label`. | `test_tool_approval_middleware_standing_rules_include_hosted_server_boundary` | +| Hosted server boundary | `function_call.hosted` determines provider execution. Standing approval compares its persisted `server_label` with the call's `server_name` and never crosses that boundary. | `test_tool_approval_middleware_standing_rules_include_hosted_server_boundary` | | Argument-scoped rule | Exact arguments are required; empty arguments are not tool-wide. | `test_tool_approval_middleware_always_approve_tool_with_arguments_rule`, `test_tool_approval_middleware_empty_arguments_rule_is_not_tool_wide` | | Provider-injected approval tool | A tool added during `before_run` defers to in-run resolution, executes once, and emits one result. | `packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_agent_approval_deferred_provider_tool_executes` | | AG-UI provider boundary | Completed local approval controls from AG-UI request and snapshot replay are absent from raw chat-client input while deferred and hosted approvals keep their respective in-run/provider paths. Lifecycle-authenticated local approvals preserve current request-generation authority through occurrence rebinding; authenticated rejection/cancellation cleans only the matching fixed provider-scoped policy record, while malformed or unauthorized controls clean nothing. | `packages/ag-ui/tests/ag_ui/test_endpoint.py::test_endpoint_does_not_forward_resolved_local_approval_control_to_chat_client`, `test_endpoint_agent_approval_deferred_provider_tool_executes`, `test_endpoint_canonical_resume_preserves_hosted_approval_for_provider`, `test_endpoint_fides_non_grant_cleans_authenticated_fixed_scope_only`, `test_endpoint_fides_approval_uses_lifecycle_bound_request_generation`, `packages/ag-ui/tests/ag_ui/test_run.py::test_filter_local_approval_responses_for_provider_removes_duplicate_completed_controls`, `test_filter_local_approval_responses_for_provider_pairs_reused_call_ids_by_occurrence`, `test_canonical_hosted_approval_resume_rejects_edited_arguments_without_mutating_pending` | diff --git a/docs/specs/005-python-content-identity-and-protocol-metadata.md b/docs/specs/005-python-content-identity-and-protocol-metadata.md new file mode 100644 index 00000000000..040c7528bb5 --- /dev/null +++ b/docs/specs/005-python-content-identity-and-protocol-metadata.md @@ -0,0 +1,256 @@ +--- +status: approved +contact: eavanvalkenburg +deciders: eavanvalkenburg +--- + +# Python content identity and protocol metadata contract + +## Scope + +This specification defines the serialized Python contract selected by +[ADR 0044](../decisions/0044-python-content-identity-and-protocol-metadata.md). It applies to `Content`, `Message`, +streamed response aggregation, and all Python provider and protocol adapters that emit or consume these values, +including OpenAI, Foundry, Anthropic, Bedrock, Gemini, Mistral, Ollama, Claude, GitHub Copilot, Copilot Studio, +Hosting Responses, Hosting A2A, DevUI, AG-UI, and A2A. + +The contract has four parts: + +1. Message identity, Agent Framework content occurrence identity, provider item identity, and call correlation remain + separate. +2. Execution routing and stream merge intent use first-class fields. +3. Provider-neutral and provider-owned metadata use one-level namespaces. +4. Legacy locations remain readable; replay-critical writers temporarily dual-write. + +This specification is normative. Package documentation may describe individual keys but must not define conflicting +locations or merge behavior. + +## First-class fields + +### Function-call and result identity + +| Field | Content types | Meaning | +| --- | --- | --- | +| `Content.id` | `function_call` | Agent Framework identity for one actionable occurrence. It is never an OpenAI or other provider item ID. | +| `Content.id` | `function_result` | The function-call occurrence settled by the result, when known. | +| `Content.call_id` | Calls and results | Provider/service call-result correlation. It may be reused in a later occurrence. | +| `Content.id` | Reasoning, computer use, hosted approval controls | Existing documented provider-identity exception. New content types do not create more exceptions. | + +An occurrence-identified Host result binds to that exact occurrence. An id-less Host result is a compatibility form +and binds only when the active batch has one unambiguous unanswered occurrence with the same `call_id`. + +### Message identity and metadata + +`Message.message_id`, `ChatResponseUpdate.message_id`, and `AgentResponseUpdate.message_id` identify the containing +provider or protocol message. They do not identify a `Content` item and must not be substituted for `Content.id` or +`call_id`. + +When streamed updates carry a non-empty `message_id`, aggregation preserves it on the finalized `Message`. A change +between two non-empty message IDs starts a new message; an absent ID keeps the legacy role-and-adjacency grouping. +Adapters preserve a provider message ID when the protocol exposes one. Deserializing an older message without an ID +does not manufacture one. + +Message- and update-level `additional_properties` follow the same namespace rules as content metadata. A protocol's +message metadata belongs in that protocol owner's namespace; content item metadata remains on the relevant `Content`. + +`ChatResponseUpdate.merge_mode` and `AgentResponseUpdate.merge_mode` apply to the containing logical message. An +incoming `replace` update clears the accumulated contents of the message selected by `message_id` and role before +the update's contents are applied. `append` and `None` retain the existing accumulation behavior. Protocols must use +update-level replacement for authoritative heterogeneous message or artifact snapshots whose prior URI, binary, or +other non-mergeable content must be discarded. + +### Hosted execution + +`function_call` content serializes: + +```json +{ + "type": "function_call", + "call_id": "call_123", + "id": "occurrence_123", + "hosted": true, + "server_name": "mcp_server" +} +``` + +- `hosted=True` means the provider or service executes the call. Agent Framework must not execute it locally. +- `hosted=False` means the call is locally actionable. +- `server_name` is the descriptive hosted-server boundary. It does not decide routing. +- `hosted` is valid only on `function_call` content and must be a boolean when supplied. Construction normalizes an + omitted value to `False` unless the legacy migration below infers `True`; serialization always writes a boolean. +- A streamed call consists of complete `Content` fragments; the wire does not split or incrementally deserialize the + boolean itself. Fragments can still differ because argument-only deltas may omit hosted ownership and therefore + normalize to `False`. Aggregation uses logical OR so a fragment that established `hosted=True` cannot later become + locally actionable. Conflicting non-empty `server_name` values are rejected. + +Legacy `additional_properties["server_label"]` remains readable. When first-class `hosted` is absent, a truthy +string `server_label` infers `hosted=True`, fills a missing `server_name`, and emits `FutureWarning`. This compatibility +inference uses only the legacy `server_label`; first-class `server_name` never implies hosted ownership. When +`hosted` and legacy `server_label` are both absent, `hosted` becomes `False` even if `server_name` is present. When +both server names are present they must match. The same migration runs through construction, `from_dict()`, nested +approval content, and pickle restoration. + +`ToolApprovalRule.server_label` remains public API and persisted state. New rules source its value from +`function_call.server_name`. + +### Stream merge intent + +The following content types accept +`merge_mode: Literal["append", "replace"] | None`: + +- `text` +- `text_reasoning` +- `function_call` +- `usage` +- `code_interpreter_tool_call` +- `code_interpreter_tool_result` + +URI, data, image, audio, and other non-mergeable content types are atomic values: adapters emit a complete item +rather than fragments of a URL or base64 payload. They therefore do not accept content-level `merge_mode`. When an +authoritative protocol snapshot replaces a heterogeneous message containing those items, the adapter uses the +update-level replacement described above. + +The mode belongs to the incoming fragment: + +- `append` applies the content type's incremental merge. +- `replace` makes the incoming semantic payload authoritative after identity validation. +- `None` preserves the legacy type-specific behavior. + +`None` is not an alias for `replace`, and there is no single universal implicit default. It delegates to the behavior +that existed before `merge_mode`: text and reasoning text concatenate, function-call fields and arguments merge +incrementally, usage values add, and code-interpreter inputs and outputs use their existing field-wise list merge. +This compatibility mode exists for older adapters; new provider updates must state `append` or `replace` explicitly. + +Content-level modes are serialized on individual content items. Update-level modes are serialized on +`ChatResponseUpdate` and `AgentResponseUpdate`. `ChatResponse.from_updates()`, +`AgentResponse.from_updates()`, and `ResponseStream.get_final_response()` consume update-level modes and recursively +clear content-level modes from finalized content, including nested content items. + +Providers must mark every mergeable streamed item they emit. Delta events use `append`; authoritative payload +snapshots use `replace`. An event name such as `*.done` does not by itself imply replacement when the event carries +only final metadata and omits an earlier streamed payload. Unreleased `stream_complete` and `complete` metadata +markers are not compatibility formats. + +## Metadata namespaces + +`additional_properties` reserves: + +- `core`: provider-neutral semantics documented by core; +- a provider/protocol owner name such as `openai`, `foundry`, or `a2a`; +- leading-underscore top-level keys for core-private state. + +For a package-owned namespace, the canonical key is the normalized integration package suffix: remove the +`agent-framework-` prefix and replace hyphens with underscores, for example `agent-framework-ag-ui` uses `ag_ui` and +`agent-framework-github-copilot` uses `github_copilot`. A package implementing another owner's wire protocol uses that +protocol owner's namespace instead and documents the exception; for example Foundry and hosting packages use +`openai` for Responses metadata and `foundry` only for Foundry-owned data. + +Namespace values are mappings with one level of keys. Each value must be plain serializable data. A value may itself +be a JSON mapping or list, but namespace owners do not add a second namespace layer. + +Example: + +```json +{ + "core": {"model_output_kind": "refusal"}, + "openai": {"item_id": "fc_123", "image_detail": "high"}, + "_approval_request_id": "approval_123", + "application_ticket": "ABC-42" +} +``` + +### Exact mapping + +| Owner | Content/message | Legacy location | Canonical location | Compatibility | +| --- | --- | --- | --- | --- | +| core | refusal text | `model_output_kind` | `core["model_output_kind"]` | Read indefinitely; dual-write through the writer package's next major version. | +| core/OpenAI | input content | `prompt_cache_breakpoint` | `core["prompt_cache_breakpoint"]` | Read indefinitely; dual-write through the OpenAI package's next major version. | +| OpenAI | function call/result and provider output items | `fc_id` or `item_id` | `openai["item_id"]` | Read indefinitely; dual-write when persisted replay or an independently versioned host consumes the value. | +| OpenAI | input image/file discriminator | `openai_content_type` | `openai["content_type"]` | Read indefinitely; dual-write through the writer package's next major version. | +| OpenAI | image input/screenshot | `detail` | `openai["image_detail"]` | Read indefinitely; dual-write through the writer package's next major version. | +| OpenAI | computer call | `computer_action_format` | `openai["computer_action_format"]` | Read indefinitely; dual-write through the writer package's next major version. | +| OpenAI | text output | `logprobs` | `openai["logprobs"]` | Read indefinitely; dual-write when the content can cross package boundaries. | +| OpenAI | reasoning summary marker | `reasoning_text` | `openai["reasoning_text"]` | Read indefinitely; dual-write for persisted replay. | +| OpenAI | Chat Completions message | `reasoning_details` | `openai["reasoning_details"]` | Read indefinitely. Provider-emitted opaque reasoning payloads use `Content.protected_data`. | +| OpenAI | shell tool declaration | `openai.responses.shell.environment` | `openai["shell_environment"]` | Read indefinitely; dual-write for replay. | +| OpenAI | shell result | `openai.responses.shell.output_type` | `openai["shell_output_type"]` | Read indefinitely; dual-write for replay. | +| OpenAI | local shell result | `openai.responses.local_shell.call_item_id` | `openai["local_shell_call_item_id"]` | Read indefinitely; dual-write for replay. | +| OpenAI | local shell call/result | `openai.local_shell_command_parts` | `openai["local_shell_command_parts"]` | Read indefinitely; dual-write for replay. | +| OpenAI | OAuth consent request server | `server_label` | `openai["server_label"]` | Read indefinitely; dual-write through the writer package's next major version. | +| Foundry | reasoning content | `__foundry_reasoning_replay_item__` | `foundry["reasoning_item"]` | Read indefinitely; dual-write through the Foundry package's next major version. | +| Bedrock | redacted reasoning marker | `bedrock.redacted_reasoning` | `bedrock["redacted_reasoning"]` | Read indefinitely; dual-write through the Bedrock package's next major version. | +| Mistral | response service tier | `service_tier` | `mistral["service_tier"]` | Read indefinitely; dual-write through the Mistral package's next major version. | +| A2A | response/update/message metadata | `a2a_metadata` | `a2a` | Read indefinitely with a warning for legacy-only input; dual-write through the A2A package's next major version. | +| all | data/URI/hosted-file display name | `filename` | `Content.name` | Read indefinitely where accepted; dual-write where older adapters require it. | +| all | data/URI provider file identity | `file_id` | `Content.file_id` | Read indefinitely where accepted; dual-write where older adapters require it. | +| all | function/reasoning/tool status | `additional_properties["status"]` | `Content.status` | Read indefinitely where accepted; new writers use the field. | +| hosted adapters | hosted routing | `server_label` | `Content.hosted` plus `Content.server_name` | Constructor migration as defined above; replay-critical writers dual-write. | + +OpenAI reasoning item IDs, computer call/result IDs, and hosted approval request/response IDs retain their documented +`Content.id` exception. The table does not move core-private keys, security labels, or application metadata. + +## Merge and copy behavior + +Core exposes: + +```python +get_namespaced_value(additional_properties, namespace, key, default=None) +with_namespaced_value(additional_properties, namespace, key, value) +merge_additional_properties(existing, incoming, *, incoming_wins=True) +``` + +- Constructors copy the top-level mapping and every top-level mapping value. +- Writers replace a namespace with a copied mapping; they do not mutate a shared nested mapping. +- When both sides contain mappings under the same top-level key, merging combines those mappings one level deep. +- Other values remain opaque. +- Direct content addition keeps existing values on collision. +- Response, workflow response, function-result carrier, and code-interpreter aggregation keep incoming values. +- The rule applies to application-owned top-level mappings as well as reserved namespaces. + +## Replay and protected data + +- Replay must not depend on `raw_representation`; it is absent after persisted history reload. +- Replay-critical namespace values contain only the minimum plain data needed to reconstruct the provider item. +- Opaque provider reasoning payloads use `Content.protected_data` when that field applies. +- Compaction token estimation removes `encrypted_content` recursively, including inside provider namespace values. +- Adapters reject malformed replay metadata instead of silently emitting a different provider transcript. + +## Version compatibility + +- Upgrades read old histories. +- Downgrades across this change are unsupported because older runtimes reject serialized `hosted` and `merge_mode`. +- Legacy reads remain until a separate breaking decision removes them. +- Replay-critical writers dual-write until that package's next major version. Display-only or request-local keys may + switch immediately. +- The owning integration package centralizes its canonical and legacy keys and may warn when it consumes legacy-only + input. Generic core namespace helpers do not contain provider-specific migration tables. +- A package that writes namespaces requires a core version that implements one-level namespace merging. + +## Authoritative validation + +| Contract | Authoritative tests | +| --- | --- | +| Field validation, serialization, legacy hosted inference, namespace helpers, one-level merge rules, merge modes, and finalization | `python/packages/core/tests/core/test_types.py` | +| Compaction strips opaque nested values | `python/packages/core/tests/core/test_compaction.py` | +| Occurrence-aware approval and Host-result matching | Scenario rows in [spec 004](004-python-function-calling-loop.md), primarily `python/packages/core/tests/core/test_function_invocation_logic.py` | +| OpenAI Responses mappings, done snapshots, hosted approvals, shell replay, and persisted-history replay | `python/packages/openai/tests/openai/test_openai_chat_client.py` | +| OpenAI Chat Completions generic/OpenAI metadata and streamed merge modes | `python/packages/openai/tests/openai/test_openai_chat_completion_client.py` | +| Foundry reasoning replay | `python/packages/foundry/tests/foundry/test_foundry_chat_client.py` | +| Foundry hosting conversion and hosted routing | `python/packages/foundry_hosting/tests/test_responses.py` | +| Hosting Responses parsing/rendering | `python/packages/hosting-responses/tests/hosting_responses/test_parsing.py` | +| DevUI Responses conversion | `python/packages/devui/tests/devui/test_conversations.py`, `python/packages/devui/tests/devui/test_mapper.py` | +| AG-UI stream merge modes, hosted/local ownership, and approval replay | `python/packages/ag-ui/tests/ag_ui/test_event_converters.py`, `test_run.py`, `test_workflow_run.py`, and `test_endpoint.py` | +| A2A stream merge modes and namespaced metadata round trip | `python/packages/a2a/tests/test_a2a_agent.py` | +| Hosting A2A message metadata conversion | `python/packages/hosting-a2a/tests/hosting_a2a/test_conversion.py` | +| Anthropic streamed merge modes and provider-hosted tool routing | `python/packages/anthropic/tests/test_anthropic_client.py` | +| Bedrock streamed merge modes and redacted-reasoning replay metadata | `python/packages/bedrock/tests/test_bedrock_client.py` | +| Gemini streamed merge modes and persisted provider-hosted tool replay | `python/packages/gemini/tests/test_gemini_client.py` | +| Mistral streamed merge modes and response metadata | `python/packages/mistral/tests/mistral/test_mistral_chat_client.py` | +| Ollama streamed merge modes | `python/packages/ollama/tests/test_ollama_chat_client.py` | +| Claude managed-agent streamed merge modes | `python/packages/claude/tests/test_claude_agent.py` | +| GitHub Copilot streamed merge modes and provider-hosted tool routing | `python/packages/github_copilot/tests/test_github_copilot_agent.py` | +| Copilot Studio streamed merge modes | `python/packages/copilotstudio/tests/test_copilot_agent.py` | + +Every changed package must test legacy-only, dual-written, and canonical-only inputs for replay-critical mappings. +Streaming tests must cover repetitive fragments and snapshots whose text begins with the accumulated prefix, so tests +verify explicit merge intent rather than a prefix heuristic. diff --git a/python/packages/a2a/AGENTS.md b/python/packages/a2a/AGENTS.md index 46f07333bd5..fb2c071354a 100644 --- a/python/packages/a2a/AGENTS.md +++ b/python/packages/a2a/AGENTS.md @@ -9,6 +9,11 @@ Agent-to-Agent (A2A) protocol support for inter-agent communication. - **`A2AServiceSessionId`** - Typed durable A2A continuation state shape stored in `AgentSession.service_session_id`. - **`A2AAgentSession`** - Deprecated compatibility session wrapper; prefer `AgentSession` + `A2AServiceSessionId`. +A2A protocol metadata is stored under `additional_properties["a2a"]`. The legacy `a2a_metadata` key remains readable +and is dual-written during the compatibility period defined by +[spec 005](../../../docs/specs/005-python-content-identity-and-protocol-metadata.md). Consuming legacy-only input +emits `FutureWarning`; new callers should write only the canonical `a2a` key. + ## Usage ### A2AAgent (Client) diff --git a/python/packages/a2a/agent_framework_a2a/_agent.py b/python/packages/a2a/agent_framework_a2a/_agent.py index f16e263cbf8..1030f6456b6 100644 --- a/python/packages/a2a/agent_framework_a2a/_agent.py +++ b/python/packages/a2a/agent_framework_a2a/_agent.py @@ -59,6 +59,9 @@ else: from typing_extensions import TypedDict # pragma: no cover +_A2A_METADATA_KEY: Final = "a2a" +_LEGACY_A2A_METADATA_KEY: Final = "a2a_metadata" + class A2AServiceSessionId(TypedDict): """Durable A2A continuation state stored in ``AgentSession.service_session_id``.""" @@ -77,6 +80,30 @@ def _with_operation_state( return update +def _a2a_metadata_properties(metadata: Mapping[str, Any] | None) -> dict[str, Any] | None: + """Return canonical and compatibility A2A metadata entries.""" + if not metadata: + return None + return { + _A2A_METADATA_KEY: metadata, + _LEGACY_A2A_METADATA_KEY: metadata, + } + + +def _read_a2a_metadata(additional_properties: Mapping[str, Any]) -> Any: + """Read canonical A2A metadata, warning only for legacy-only input.""" + if _A2A_METADATA_KEY in additional_properties: + return additional_properties[_A2A_METADATA_KEY] + if _LEGACY_A2A_METADATA_KEY not in additional_properties: + return None + warnings.warn( + "additional_properties['a2a_metadata'] is deprecated; use additional_properties['a2a'] instead.", + FutureWarning, + stacklevel=3, # Skip this helper and the adapter conversion frame. + ) + return additional_properties[_LEGACY_A2A_METADATA_KEY] + + class A2AAgentSession(AgentSession): """Session for A2A-based agents. @@ -685,14 +712,15 @@ async def _map_a2a_stream( self._validate_context(session, msg.context_id) if msg.context_id: last_context_id = msg.context_id - contents = self._parse_contents_from_a2a(msg.parts) + contents = self._parse_contents_from_a2a(msg.parts, merge_mode="replace") metadata = MessageToDict(msg.metadata) if msg.metadata else None update = AgentResponseUpdate( contents=contents, role="assistant" if msg.role == A2ARole.ROLE_AGENT else "user", response_id=msg.message_id or str(uuid.uuid4()), message_id=msg.message_id, - additional_properties={"a2a_metadata": metadata} if metadata else None, + merge_mode="replace", + additional_properties=_a2a_metadata_properties(metadata), raw_representation=msg, ) all_updates.append(update) @@ -893,7 +921,7 @@ def _updates_from_task( role="assistant" if message is None or message.role == A2ARole.ROLE_AGENT else "user", response_id=task.id, continuation_token=self._build_continuation_token(task) if background else None, - additional_properties={"a2a_metadata": task_metadata} if task_metadata else None, + additional_properties=_a2a_metadata_properties(task_metadata), raw_representation=task, ) return [_with_operation_state(update, "in_progress")] if background else [update] @@ -913,9 +941,10 @@ def _updates_from_task( role=message.role, response_id=task.id, message_id=getattr(message.raw_representation, "artifact_id", None), - additional_properties={"a2a_metadata": merged} - if (merged := {**message.additional_properties, **(task_metadata or {})}) - else None, + additional_properties=_a2a_metadata_properties({ + **message.additional_properties, + **(task_metadata or {}), + }), raw_representation=task, ) for message in task_messages @@ -931,7 +960,7 @@ def _updates_from_task( contents=[], role="assistant", response_id=task.id, - additional_properties={"a2a_metadata": task_metadata} if task_metadata else None, + additional_properties=_a2a_metadata_properties(task_metadata), raw_representation=task, ), "terminal", @@ -943,7 +972,7 @@ def _updates_from_task( contents=[], role="assistant", response_id=task.id, - additional_properties={"a2a_metadata": task_metadata} if task_metadata else None, + additional_properties=_a2a_metadata_properties(task_metadata), raw_representation=task, ), "terminal", @@ -959,7 +988,7 @@ def _updates_from_task( role="assistant", response_id=task.id, continuation_token=token, - additional_properties={"a2a_metadata": task_metadata} if task_metadata else None, + additional_properties=_a2a_metadata_properties(task_metadata), raw_representation=task, ), "in_progress", @@ -973,7 +1002,7 @@ def _updates_from_task( and status.HasField("message") and status.message.parts ): - contents = self._parse_contents_from_a2a(status.message.parts) + contents = self._parse_contents_from_a2a(status.message.parts, merge_mode="replace") if contents: return [ _with_operation_state( @@ -981,7 +1010,7 @@ def _updates_from_task( contents=contents, role="assistant" if status.message.role == A2ARole.ROLE_AGENT else "user", response_id=task.id, - additional_properties={"a2a_metadata": task_metadata} if task_metadata else None, + additional_properties=_a2a_metadata_properties(task_metadata), raw_representation=task, ), "in_progress", @@ -991,7 +1020,7 @@ def _updates_from_task( contents=contents, role="assistant" if status.message.role == A2ARole.ROLE_AGENT else "user", response_id=task.id, - additional_properties={"a2a_metadata": task_metadata} if task_metadata else None, + additional_properties=_a2a_metadata_properties(task_metadata), raw_representation=task, ) ] @@ -1007,7 +1036,10 @@ def _updates_from_task_update_event( ) -> list[AgentResponseUpdate]: """Convert A2A task update events into streaming AgentResponseUpdates.""" if isinstance(update_event, TaskArtifactUpdateEvent): - contents = self._parse_contents_from_a2a(update_event.artifact.parts) + contents = self._parse_contents_from_a2a( + update_event.artifact.parts, + merge_mode="append" if update_event.append else "replace", + ) if not contents: return [] artifact_meta = MessageToDict(update_event.artifact.metadata) if update_event.artifact.metadata else {} @@ -1018,7 +1050,8 @@ def _updates_from_task_update_event( role="assistant", response_id=update_event.task_id, message_id=update_event.artifact.artifact_id, - additional_properties={"a2a_metadata": merged_metadata} if merged_metadata else None, + merge_mode="append" if update_event.append else "replace", + additional_properties=_a2a_metadata_properties(merged_metadata), raw_representation=update_event, ) return [_with_operation_state(update, "in_progress")] if background else [update] @@ -1054,7 +1087,7 @@ def _updates_from_task_update_event( response_id=update_event.task_id, message_id=message.message_id if message is not None else None, continuation_token=continuation_token, - additional_properties={"a2a_metadata": merged_metadata} if merged_metadata else None, + additional_properties=_a2a_metadata_properties(merged_metadata), raw_representation=update_event, ) return [_with_operation_state(update, "in_progress")] if background else [update] @@ -1073,7 +1106,7 @@ def _updates_from_task_update_event( role="assistant", response_id=update_event.task_id, continuation_token=continuation_token, - additional_properties={"a2a_metadata": event_meta} if event_meta else None, + additional_properties=_a2a_metadata_properties(event_meta), raw_representation=update_event, ), "terminal", @@ -1081,7 +1114,7 @@ def _updates_from_task_update_event( ] message = update_event.status.message - contents = self._parse_contents_from_a2a(message.parts) + contents = self._parse_contents_from_a2a(message.parts, merge_mode="replace") if not contents: if not background: return [] @@ -1093,7 +1126,7 @@ def _updates_from_task_update_event( response_id=update_event.task_id, message_id=message.message_id, continuation_token=continuation_token, - additional_properties={"a2a_metadata": event_meta} if event_meta else None, + additional_properties=_a2a_metadata_properties(event_meta), raw_representation=update_event, ), "terminal", @@ -1109,8 +1142,9 @@ def _updates_from_task_update_event( role="assistant" if message.role == A2ARole.ROLE_AGENT else "user", response_id=update_event.task_id, message_id=message.message_id, + merge_mode="replace", continuation_token=continuation_token, - additional_properties={"a2a_metadata": merged_metadata} if merged_metadata else None, + additional_properties=_a2a_metadata_properties(merged_metadata), raw_representation=update_event, ), "terminal", @@ -1258,7 +1292,7 @@ def _prepare_message_for_a2a(self, message: Message, *, session: AgentSession | case _: raise ValueError(f"Unknown content type: {content.type}") - a2a_metadata = message.additional_properties.get("a2a_metadata") + a2a_metadata = _read_a2a_metadata(message.additional_properties) a2a_message = A2AMessage( role=A2ARole.ROLE_USER, @@ -1278,13 +1312,19 @@ def _prepare_message_for_a2a(self, message: Message, *, session: AgentSession | return a2a_message - def _parse_contents_from_a2a(self, parts: Sequence[A2APart]) -> list[Content]: + def _parse_contents_from_a2a( + self, + parts: Sequence[A2APart], + *, + merge_mode: Literal["append", "replace"] | None = None, + ) -> list[Content]: """Parse A2A Parts into Agent Framework Content. Transforms A2A protocol Parts into framework-native Content objects, handling text, url, raw, and data parts with metadata preservation. """ contents: list[Content] = [] + next_text_merge_mode = merge_mode for part in parts: part_metadata = MessageToDict(part.metadata) if part.metadata else None content_type = part.WhichOneof("content") @@ -1293,10 +1333,13 @@ def _parse_contents_from_a2a(self, parts: Sequence[A2APart]) -> list[Content]: contents.append( Content.from_text( text=part.text, + merge_mode=next_text_merge_mode, additional_properties=part_metadata, raw_representation=part, ) ) + if next_text_merge_mode == "replace": + next_text_merge_mode = "append" case "url": contents.append( Content.from_uri( @@ -1321,10 +1364,13 @@ def _parse_contents_from_a2a(self, parts: Sequence[A2APart]) -> list[Content]: contents.append( Content.from_text( text=MessageToJson(part.data), + merge_mode=next_text_merge_mode, additional_properties=part_metadata, raw_representation=part, ) ) + if next_text_merge_mode == "replace": + next_text_merge_mode = "append" case _: raise ValueError(f"Unknown Part content type: {content_type}") return contents diff --git a/python/packages/a2a/tests/test_a2a_agent.py b/python/packages/a2a/tests/test_a2a_agent.py index bc6c85e285e..81f3e271a40 100644 --- a/python/packages/a2a/tests/test_a2a_agent.py +++ b/python/packages/a2a/tests/test_a2a_agent.py @@ -1,6 +1,7 @@ # Copyright (c) Microsoft. All rights reserved. import json +import warnings from collections.abc import AsyncIterator from typing import Any, cast from unittest.mock import AsyncMock, MagicMock, patch @@ -443,6 +444,43 @@ def test_parse_contents_from_a2a_conversion(a2a_agent: A2AAgent) -> None: assert contents[1].text == "Second part" +def test_parse_snapshot_contents_replaces_once_then_appends_text_parts(a2a_agent: A2AAgent) -> None: + contents = a2a_agent._parse_contents_from_a2a( + [Part(text="Choose "), Part(text="blue")], + merge_mode="replace", + ) + + assert [content.merge_mode for content in contents] == ["replace", "append"] + + response = AgentResponse.from_updates([ + AgentResponseUpdate(contents=[Content.from_text("old")], role="assistant"), + AgentResponseUpdate(contents=contents, role="assistant"), + ]) + + assert response.text == "Choose blue" + + +@mark.parametrize("data_first", [False, True], ids=["text-data", "data-text"]) +def test_parse_snapshot_contents_replaces_once_across_text_and_data_parts( + a2a_agent: A2AAgent, *, data_first: bool +) -> None: + from google.protobuf.json_format import ParseDict + from google.protobuf.struct_pb2 import Value # ty: ignore[unresolved-import] + + data_part = Part(data=ParseDict({"choice": "blue"}, Value())) + parts = [data_part, Part(text=" selected")] if data_first else [Part(text="Selected "), data_part] + contents = a2a_agent._parse_contents_from_a2a(parts, merge_mode="replace") + + assert [content.merge_mode for content in contents] == ["replace", "append"] + + response = AgentResponse.from_updates([ + AgentResponseUpdate(contents=[Content.from_text("old")], role="assistant"), + AgentResponseUpdate(contents=contents, role="assistant"), + ]) + + assert response.text == "".join(content.text or "" for content in contents) + + def test_prepare_message_for_a2a_with_error_content(a2a_agent: A2AAgent) -> None: """Test _prepare_message_for_a2a with ErrorContent.""" @@ -596,7 +634,7 @@ def test_prepare_message_for_a2a_forwards_context_id() -> None: message = Message( role="user", contents=[Content.from_text(text="Continue the task")], - additional_properties={"a2a_metadata": {"trace_id": "trace-456"}}, + additional_properties={"a2a": {"trace_id": "trace-456"}}, ) session = A2AAgentSession(context_id="ctx-123") @@ -606,6 +644,41 @@ def test_prepare_message_for_a2a_forwards_context_id() -> None: assert result.metadata == {"trace_id": "trace-456"} +def test_prepare_message_for_a2a_warns_for_legacy_metadata() -> None: + """Legacy-only A2A metadata remains readable but directs callers to the canonical namespace.""" + agent = A2AAgent(client=MagicMock(), http_client=None) + message = Message( + role="user", + contents=[Content.from_text(text="Continue the task")], + additional_properties={"a2a_metadata": {"trace_id": "trace-legacy"}}, + ) + + with warns(FutureWarning, match=r"a2a_metadata.*deprecated"): + result = agent._prepare_message_for_a2a(message) + + assert result.metadata == {"trace_id": "trace-legacy"} + + +def test_prepare_message_for_a2a_prefers_canonical_metadata() -> None: + """Canonical metadata wins during the dual-written compatibility period.""" + agent = A2AAgent(client=MagicMock(), http_client=None) + message = Message( + role="user", + contents=[Content.from_text(text="Continue the task")], + additional_properties={ + "a2a": {"trace_id": "trace-canonical"}, + "a2a_metadata": {"trace_id": "trace-legacy"}, + }, + ) + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + result = agent._prepare_message_for_a2a(message) + + assert result.metadata == {"trace_id": "trace-canonical"} + assert not [warning for warning in caught if issubclass(warning.category, FutureWarning)] + + def test_prepare_message_for_a2a_uses_fallback_context_id() -> None: """Test that service_session_id from a plain session is used when message has no context_id property.""" @@ -1735,6 +1808,46 @@ async def test_streaming_artifact_update_event_yields_content( assert updates[0].raw_representation == update_event +@mark.parametrize( + ("replacement_part", "expected_type"), + [ + (Part(url="https://example.com/replacement", media_type="text/plain"), "uri"), + (Part(raw=b"replacement", media_type="application/octet-stream"), "data"), + ], + ids=["uri", "raw"], +) +async def test_streaming_artifact_replacement_discards_previous_heterogeneous_snapshot( + a2a_agent: A2AAgent, + mock_a2a_client: MockA2AClient, + replacement_part: Part, + expected_type: str, +) -> None: + first = TaskArtifactUpdateEvent( + task_id="task-art-replace", + context_id="ctx-art-replace", + artifact=Artifact(artifact_id="artifact-replace", parts=[Part(text="stale")]), + append=False, + ) + replacement = TaskArtifactUpdateEvent( + task_id="task-art-replace", + context_id="ctx-art-replace", + artifact=Artifact(artifact_id="artifact-replace", parts=[replacement_part]), + append=False, + ) + mock_a2a_client.responses.extend([ + StreamResponse(artifact_update=first), + StreamResponse(artifact_update=replacement), + ]) + + stream = a2a_agent.run("Hello", stream=True) + updates = [update async for update in stream] + response = await stream.get_final_response() + + assert [update.merge_mode for update in updates] == ["replace", "replace"] + assert len(response.messages) == 1 + assert [content.type for content in response.messages[0].contents] == [expected_type] + + async def test_streaming_status_update_event_yields_content( a2a_agent: A2AAgent, mock_a2a_client: MockA2AClient ) -> None: @@ -1977,6 +2090,8 @@ async def test_streaming_artifact_update_event_does_not_duplicate_terminal_task_ updates: list[AgentResponseUpdate] = [] async for update in stream: updates.append(update) + assert updates[0].contents[0].merge_mode == "replace" + assert updates[1].contents[0].merge_mode == "append" response = await stream.get_final_response() assert [update.text for update in updates] == ["Hello ", "world"] @@ -2137,6 +2252,8 @@ async def test_message_metadata_propagated(a2a_agent: A2AAgent, mock_a2a_client: mock_a2a_client.responses.append(StreamResponse(message=msg)) response = await a2a_agent.run("hello") + assert response.additional_properties["a2a"]["source"] == "server" + assert response.additional_properties["a2a"]["trace_id"] == "abc" assert response.additional_properties["a2a_metadata"]["source"] == "server" assert response.additional_properties["a2a_metadata"]["trace_id"] == "abc" @@ -2210,6 +2327,10 @@ async def test_task_artifact_update_event_metadata_merged(a2a_agent: A2AAgent, m artifact_update = updates[0] assert artifact_update.additional_properties is not None + assert artifact_update.additional_properties["a2a"] == { + "from_artifact": True, + "from_event": True, + } metadata = artifact_update.additional_properties["a2a_metadata"] assert metadata["from_artifact"] is True assert metadata["from_event"] is True diff --git a/python/packages/ag-ui/AGENTS.md b/python/packages/ag-ui/AGENTS.md index 7990f963703..4470067c1ba 100644 --- a/python/packages/ag-ui/AGENTS.md +++ b/python/packages/ag-ui/AGENTS.md @@ -99,6 +99,9 @@ AG-UI protocol integration for building agent UIs with the AG-UI standard. remains the provider/service `call_id`. Hosted approvals preserve the provider-issued approval request id. The lifecycle stores these identities separately so resume responses carry the authoritative approval occurrence id without rewriting tool-result correlation ids. +- Hosted/local execution ownership comes from `function_call.hosted`; `server_name` supplies the hosted-server + boundary persisted as `ApprovalOccurrence.server_label`. Legacy `additional_properties["server_label"]` remains a + compatibility input, not the primary routing signal. - Default stateless conversation history is client-controlled, including historical tool calls and results. Never document conversational tool results as authorization or policy evidence; use deterministic server-side checks, server-validated approvals, or scoped authoritative snapshots. diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py b/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py index 11004a75cb8..5d2eabb2736 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py @@ -124,6 +124,7 @@ _AGUI_MCP_TOOL_RESULT_KEY, _approval_interrupt_id, _bound_host_payload_history, + _function_call_is_hosted, _function_call_server_label, _mcp_host_history_fields, _model_items_for_agui_replay, @@ -799,7 +800,7 @@ def _function_call_execution_owner( has_deferred_owner: bool = False, ) -> ApprovalExecutionOwner: """Resolve execution ownership only after the call and available tools exist.""" - if _function_call_server_label(function_call): + if _function_call_is_hosted(function_call): return ApprovalExecutionOwner.HOSTED tool = _get_tool_map(tools).get(function_call.name) if tools and function_call.name else None if tool is not None and not getattr(tool, "declaration_only", False): @@ -1836,8 +1837,12 @@ async def _resolve_approval_responses( if primary_response.function_call is not None: if server_label: primary_response.function_call.additional_properties["server_label"] = server_label + primary_response.function_call.hosted = True + primary_response.function_call.server_name = server_label else: primary_response.function_call.additional_properties.pop("server_label", None) + primary_response.function_call.hosted = False + primary_response.function_call.server_name = None primary_response.function_call.id = pending_entry.function_call_id primary_response.additional_properties[_APPROVAL_REQUEST_ID_KEY] = ( pending_entry.response_id or pending_entry.identity.interrupt_id diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_client.py b/python/packages/ag-ui/agent_framework_ag_ui/_client.py index 90b45ff8fc7..7f21cc8c44e 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_client.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_client.py @@ -11,8 +11,7 @@ import uuid import warnings from binascii import Error as BinasciiError -from collections.abc import AsyncIterable, Awaitable, Mapping, MutableSequence, Sequence -from functools import wraps +from collections.abc import AsyncIterable, Awaitable, Mapping, Sequence from typing import TYPE_CHECKING, Any, Generic, TypedDict, cast import httpx @@ -21,7 +20,6 @@ ChatResponse, ChatResponseUpdate, Content, - FunctionTool, Message, ResponseInvalidatedException, ResponseStream, @@ -59,15 +57,6 @@ logger: logging.Logger = logging.getLogger("agent_framework.ag_ui") -def _unwrap_server_function_call_contents(contents: MutableSequence[Content | dict[str, Any]]) -> None: - """Replace server_function_call instances with their underlying call content.""" - for idx, content in enumerate(contents): - if content.type == "server_function_call": # type: ignore[union-attr] - contents[idx] = content.function_call # type: ignore[assignment, union-attr] - - -BaseChatClientT = TypeVar("BaseChatClientT", bound=type[BaseChatClient[Any]]) - AGUIChatOptionsT = TypeVar( "AGUIChatOptionsT", bound=TypedDict, # type: ignore[valid-type] @@ -124,47 +113,6 @@ def _extract_legacy_json_state(message: Message) -> dict[str, Any] | None: return _decode_json_state(content) -def _apply_server_function_call_unwrap(client: BaseChatClientT) -> BaseChatClientT: - """Class decorator that unwraps server-side function calls after tool handling.""" - - original_get_response = client.get_response - - @wraps(original_get_response) - def response_wrapper( - self, *args: Any, stream: bool = False, **kwargs: Any - ) -> Awaitable[ChatResponse] | ResponseStream[ChatResponseUpdate, ChatResponse]: - if stream: - stream_response = original_get_response(self, *args, stream=True, **kwargs) - if isinstance(stream_response, ResponseStream): - return stream_response.with_transform_hook(_map_update) - return ResponseStream(_stream_wrapper_impl(stream_response)) - return _response_wrapper_impl(self, original_get_response, *args, **kwargs) - - async def _response_wrapper_impl(self, original_func: Any, *args: Any, **kwargs: Any) -> ChatResponse: - """Non-streaming wrapper implementation.""" - response = await original_func(self, *args, stream=False, **kwargs) - if response.messages: - for message in response.messages: - _unwrap_server_function_call_contents(cast(MutableSequence[Content | dict[str, Any]], message.contents)) - return response - - async def _stream_wrapper_impl(stream: Any) -> AsyncIterable[ChatResponseUpdate]: - """Streaming wrapper implementation.""" - if isinstance(stream, Awaitable): - stream = await stream - async for update in stream: - _unwrap_server_function_call_contents(cast(MutableSequence[Content | dict[str, Any]], update.contents)) - yield update - - def _map_update(update: ChatResponseUpdate) -> ChatResponseUpdate: - _unwrap_server_function_call_contents(cast(MutableSequence[Content | dict[str, Any]], update.contents)) - return update - - client.get_response = response_wrapper # type: ignore[assignment] - return client - - -@_apply_server_function_call_unwrap class AGUIChatClient( FunctionInvocationLayer[AGUIChatOptionsT], ChatMiddlewareLayer[AGUIChatOptionsT], @@ -312,28 +260,6 @@ async def __aexit__(self, *args: Any) -> None: """Exit async context manager.""" await self.close() - def _register_server_tool_placeholder(self, tool_name: str) -> None: - """Register a declaration-only placeholder so function invocation skips execution.""" - - config = getattr(self, "function_invocation_configuration", None) - if not isinstance(config, dict): - return - additional_tools = list(config.get("additional_tools", [])) - if any(getattr(tool, "name", None) == tool_name for tool in additional_tools): - return - - placeholder: FunctionTool = FunctionTool( - name=tool_name, - description="Server-managed tool placeholder (AG-UI)", - func=None, - ) - additional_tools.append(placeholder) - config["additional_tools"] = additional_tools - registered: set[str] = getattr(self, "_registered_server_tools", set()) - registered.add(tool_name) - self._registered_server_tools = registered - logger.debug(f"[AGUIChatClient] Registered server placeholder: {tool_name}") - def _extract_state_from_messages( self, messages: Sequence[Message], @@ -530,7 +456,7 @@ async def _streaming_impl( extra={"role": update.role, "contents": [type(c).__name__ for c in update.contents]}, ) # Distinguish client vs server tools - for i, content in enumerate(update.contents): + for content in update.contents: if content.type == "function_call": logger.debug( f"[AGUIChatClient] Function call: {content.name}, in client_tool_set: {content.name in client_tool_set}" @@ -541,13 +467,7 @@ async def _streaming_impl( content.additional_properties = {} content.additional_properties["agui_thread_id"] = thread_id else: - # Server tool - wrap so function invocation ignores it - logger.debug(f"[AGUIChatClient] Wrapping server tool: {content.name}") - self._register_server_tool_placeholder(content.name) # type: ignore[arg-type] - update.contents[i] = Content( - type=cast(Any, "server_function_call"), - function_call=content, - ) + content.hosted = True yield update except GeneratorExit: diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_event_converters.py b/python/packages/ag-ui/agent_framework_ag_ui/_event_converters.py index 9d3dad2ee17..08b3614a37b 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_event_converters.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_event_converters.py @@ -190,7 +190,7 @@ def _handle_text_message_content(self, event: dict[str, Any]) -> ChatResponseUpd return ChatResponseUpdate( role="assistant", message_id=self.current_message_id, - contents=[Content.from_text(text=delta)], + contents=[Content.from_text(text=delta, merge_mode="append")], ) def _handle_text_message_end(self, event: dict[str, Any]) -> ChatResponseUpdate | None: @@ -210,6 +210,7 @@ def _handle_tool_call_start(self, event: dict[str, Any]) -> ChatResponseUpdate: call_id=self.current_tool_call_id or "", name=self.current_tool_name or "", arguments="", + merge_mode="append", ) ], ) @@ -238,6 +239,7 @@ def _handle_tool_call_args(self, event: dict[str, Any]) -> ChatResponseUpdate | call_id=self.current_tool_call_id or "", name=self.current_tool_name or "", arguments=delta, + merge_mode="append", ) ], ) @@ -350,5 +352,11 @@ def _handle_custom_event(self, event: dict[str, Any], raw_event_type: str) -> Ch else: update.message_id = message_id if annotations: - update.contents = [Content.from_text(text="", annotations=cast("list[Annotation]", annotations))] + update.contents = [ + Content.from_text( + text="", + merge_mode="append", + annotations=cast("list[Annotation]", annotations), + ) + ] return update diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_run_common.py b/python/packages/ag-ui/agent_framework_ag_ui/_run_common.py index 0f931ff88e2..73e343e57ed 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_run_common.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_run_common.py @@ -41,6 +41,7 @@ _approval_interrupt_id, _extract_mcp_tool_result_host_payload, _extract_tool_result_marker_values, + _function_call_is_hosted, _mcp_host_history_fields, _model_items_for_agui_replay, _stringify_tool_result, @@ -997,7 +998,7 @@ def _emit_approval_request( ) ) if interrupt_id: - response_schema = _approval_response_schema() if func_call.additional_properties.get("server_label") else None + response_schema = _approval_response_schema() if _function_call_is_hosted(func_call) else None flow.interrupts.append( _approval_interrupt_for_function_call( interrupt_id=str(interrupt_id), diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_utils.py b/python/packages/ag-ui/agent_framework_ag_ui/_utils.py index bd27f0b55f5..233355d27c4 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_utils.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_utils.py @@ -307,17 +307,30 @@ def _function_call_server_label(function_call: Any) -> str | None: """Return a normalized hosted-tool server label.""" if function_call is None: return None + server_name = getattr(function_call, "server_name", None) + if isinstance(server_name, str) and server_name: + return server_name server_label = getattr(function_call, "additional_properties", {}).get("server_label") return server_label if isinstance(server_label, str) and server_label else None +def _function_call_is_hosted(function_call: Any) -> bool: + """Return whether the provider, rather than Agent Framework, executes the call.""" + if function_call is None: + return False + hosted = getattr(function_call, "hosted", None) + if type(hosted) is bool: + return hosted + return _function_call_server_label(function_call) is not None + + def _approval_interrupt_id(content: Any) -> str | None: """Return the canonical client and lifecycle identity for an approval request.""" function_call = getattr(content, "function_call", None) if function_call is None: return None request_id = getattr(content, "id", None) - if _function_call_server_label(function_call) is not None: + if _function_call_is_hosted(function_call): return request_id if isinstance(request_id, str) and request_id else None if getattr(content, "additional_properties", {}).get("_replacement_approval_request") is True: return request_id if isinstance(request_id, str) and request_id else None diff --git a/python/packages/ag-ui/agent_framework_ag_ui/_workflow_run.py b/python/packages/ag-ui/agent_framework_ag_ui/_workflow_run.py index 7017931580b..fa92cd400ce 100644 --- a/python/packages/ag-ui/agent_framework_ag_ui/_workflow_run.py +++ b/python/packages/ag-ui/agent_framework_ag_ui/_workflow_run.py @@ -57,7 +57,7 @@ _normalize_resume_interrupts, _resume_contract_error, ) -from ._utils import canonical_function_arguments, generate_event_id, make_json_safe +from ._utils import _function_call_is_hosted, canonical_function_arguments, generate_event_id, make_json_safe logger = logging.getLogger(__name__) @@ -327,11 +327,7 @@ def _interrupt_entry_for_request_event(request_event: Any) -> dict[str, Any] | N ): workflow_metadata = _workflow_interrupt_metadata(request_payload, value)["agent_framework"] workflow_metadata.pop("type", None) - response_schema = ( - _approval_response_schema() - if request_data.function_call.additional_properties.get("server_label") - else None - ) + response_schema = _approval_response_schema() if _function_call_is_hosted(request_data.function_call) else None return _approval_interrupt_for_function_call( interrupt_id=str(request_payload["request_id"]), function_call=request_data.function_call, @@ -599,9 +595,9 @@ def _coerce_compact_approval_response(request_data: Content, candidate: dict[str standard_edited_arguments = candidate.get("editedArgs") if request_data.function_call is None: return None - if ( - direct_edited_arguments or standard_edited_arguments is not None - ) and request_data.function_call.additional_properties.get("server_label"): + if (direct_edited_arguments or standard_edited_arguments is not None) and _function_call_is_hosted( + request_data.function_call + ): return None original_arguments = request_data.function_call.parse_arguments() or {} @@ -630,6 +626,10 @@ def _coerce_compact_approval_response(request_data: Content, candidate: dict[str call_id=request_data.function_call.call_id or "", name=request_data.function_call.name or "", arguments=final_arguments, + id=request_data.function_call.id, + hosted=request_data.function_call.hosted, + server_name=request_data.function_call.server_name, + status=request_data.function_call.status, informational_only=request_data.function_call.informational_only, annotations=request_data.function_call.annotations, additional_properties=request_data.function_call.additional_properties, diff --git a/python/packages/ag-ui/tests/ag_ui/test_ag_ui_client.py b/python/packages/ag-ui/tests/ag_ui/test_ag_ui_client.py index 7ea74f72021..399e2202411 100644 --- a/python/packages/ag-ui/tests/ag_ui/test_ag_ui_client.py +++ b/python/packages/ag-ui/tests/ag_ui/test_ag_ui_client.py @@ -1239,8 +1239,8 @@ async def mock_post_run(*args: object, **kwargs: Any) -> AsyncGenerator[dict[str assert response is not None - async def test_server_tool_calls_unwrapped_after_invocation(self, monkeypatch: MonkeyPatch) -> None: - """Ensure server-side tool calls are exposed as FunctionCallContent after processing.""" + async def test_server_tool_calls_are_marked_hosted(self, monkeypatch: MonkeyPatch) -> None: + """Ensure server-side tool calls remain function calls and are not locally actionable.""" mock_events = [ {"type": "RUN_STARTED", "threadId": "thread_1", "runId": "run_1"}, @@ -1267,6 +1267,7 @@ async def mock_post_run(*args: object, **kwargs: Any) -> AsyncGenerator[dict[str ] assert function_calls assert function_calls[0].name == "get_time_zone" + assert function_calls[0].hosted is True assert not any(content.type == "server_function_call" for update in updates for content in update.contents) @@ -1345,15 +1346,6 @@ async def test_extract_state_from_empty_messages(self) -> None: assert result_messages == [] assert state is None - async def test_register_server_tool_non_dict_config(self) -> None: - """Non-dict function_invocation_configuration is a no-op.""" - client = StubAGUIChatClient( - endpoint="http://localhost:8888/", - function_invocation_configuration=None, # type: ignore[arg-type] - ) - # Should not raise - client._register_server_tool_placeholder("some_tool") - async def test_non_streaming_response(self, monkeypatch: MonkeyPatch) -> None: """Non-streaming path collects updates into ChatResponse.""" mock_events = [ diff --git a/python/packages/ag-ui/tests/ag_ui/test_endpoint.py b/python/packages/ag-ui/tests/ag_ui/test_endpoint.py index 1978ead021d..2a2fc812832 100644 --- a/python/packages/ag-ui/tests/ag_ui/test_endpoint.py +++ b/python/packages/ag-ui/tests/ag_ui/test_endpoint.py @@ -12367,7 +12367,8 @@ async def test_endpoint_canonical_resume_preserves_hosted_approval_for_provider( call_id=call_id, name="docs_search", arguments={"query": "azure"}, - additional_properties={"server_label": server_label}, + hosted=True, + server_name=server_label, ) def docs_search(query: str) -> str: @@ -12451,7 +12452,9 @@ async def stream_fn( assert approval_responses[0].id == call_id assert approval_responses[0].approved is True assert approval_responses[0].function_call is not None - assert approval_responses[0].function_call.additional_properties["server_label"] == server_label + assert approval_responses[0].function_call.hosted is True + assert approval_responses[0].function_call.server_name == server_label + assert "server_label" not in approval_responses[0].function_call.additional_properties retry_response = client.post( "/approval", diff --git a/python/packages/ag-ui/tests/ag_ui/test_event_converters.py b/python/packages/ag-ui/tests/ag_ui/test_event_converters.py index bacee2e7aa4..ba4be008879 100644 --- a/python/packages/ag-ui/tests/ag_ui/test_event_converters.py +++ b/python/packages/ag-ui/tests/ag_ui/test_event_converters.py @@ -65,6 +65,7 @@ def test_text_message_content_event(self) -> None: assert update.message_id == "msg_1" assert len(update.contents) == 1 assert update.contents[0].text == "Hello" + assert update.contents[0].merge_mode == "append" def test_text_message_streaming(self) -> None: """Test streaming text across multiple TEXT_MESSAGE_CONTENT events.""" @@ -112,6 +113,7 @@ def test_tool_call_start_event(self) -> None: assert update.contents[0].call_id == "call_123" assert update.contents[0].name == "get_weather" assert update.contents[0].arguments == "" + assert update.contents[0].merge_mode == "append" assert converter.current_tool_call_id == "call_123" assert converter.current_tool_name == "get_weather" @@ -161,6 +163,8 @@ def test_tool_call_args_streaming(self) -> None: assert all(update is not None for update in updates) assert updates[0].contents[0].arguments == '{"query": "' assert updates[1].contents[0].arguments == 'latest news"}' + assert updates[0].contents[0].merge_mode == "append" + assert updates[1].contents[0].merge_mode == "append" assert converter.accumulated_tool_args == '{"query": "latest news"}' def test_tool_call_end_event(self) -> None: diff --git a/python/packages/ag-ui/tests/ag_ui/test_utils.py b/python/packages/ag-ui/tests/ag_ui/test_utils.py index 57f97de8159..31d11593175 100644 --- a/python/packages/ag-ui/tests/ag_ui/test_utils.py +++ b/python/packages/ag-ui/tests/ag_ui/test_utils.py @@ -5,7 +5,10 @@ from dataclasses import dataclass from datetime import date, datetime +from agent_framework import Content + from agent_framework_ag_ui._utils import ( + _function_call_is_hosted, generate_event_id, make_json_safe, merge_state, @@ -22,6 +25,31 @@ def test_generate_event_id(): assert len(id1) > 0 +def test_function_call_hosted_flag_is_authoritative() -> None: + local_call = Content.from_function_call( + call_id="local", + name="local_tool", + hosted=False, + server_name="descriptive-boundary", + ) + hosted_call = Content.from_function_call( + call_id="hosted", + name="hosted_tool", + hosted=True, + ) + + assert _function_call_is_hosted(local_call) is False + assert _function_call_is_hosted(hosted_call) is True + + +def test_function_call_hosted_legacy_fallback() -> None: + @dataclass + class LegacyCall: + additional_properties: dict[str, str] + + assert _function_call_is_hosted(LegacyCall({"server_label": "legacy-server"})) is True + + def test_merge_state(): """Test state merging.""" current: dict[str, int] = {"a": 1, "b": 2} diff --git a/python/packages/anthropic/agent_framework_anthropic/_chat_client.py b/python/packages/anthropic/agent_framework_anthropic/_chat_client.py index 146b9f6a998..b26e6d60757 100644 --- a/python/packages/anthropic/agent_framework_anthropic/_chat_client.py +++ b/python/packages/anthropic/agent_framework_anthropic/_chat_client.py @@ -1269,14 +1269,21 @@ def _process_stream_event( usage_details: list[Content] = [] if event.message.usage and (details := self._parse_usage_from_anthropic(event.message.usage)): usage_details.append( - Content.from_usage(usage_details=self._incremental_usage(details, emitted_usage)) + Content.from_usage( + usage_details=self._incremental_usage(details, emitted_usage), + merge_mode="append", + ) ) return ChatResponseUpdate( role="assistant", response_id=event.message.id, contents=[ - *self._parse_contents_from_anthropic(event.message.content, request_state=request_state), + *self._parse_contents_from_anthropic( + event.message.content, + request_state=request_state, + streaming=True, + ), *usage_details, ], model=event.message.model, @@ -1289,6 +1296,7 @@ def _process_stream_event( contents=[ Content.from_usage( usage_details=self._incremental_usage(usage, emitted_usage), + merge_mode="append", raw_representation=event.usage, ) ] @@ -1300,13 +1308,21 @@ def _process_stream_event( case "message_stop": logger.debug("Received message_stop event; no content to process.") case "content_block_start": - contents = self._parse_contents_from_anthropic([event.content_block], request_state=request_state) + contents = self._parse_contents_from_anthropic( + [event.content_block], + request_state=request_state, + streaming=True, + ) return ChatResponseUpdate( contents=contents, raw_representation=event, ) case "content_block_delta": - contents = self._parse_contents_from_anthropic([event.delta], request_state=request_state) + contents = self._parse_contents_from_anthropic( + [event.delta], + request_state=request_state, + streaming=True, + ) return ChatResponseUpdate( contents=contents, raw_representation=event, @@ -1370,8 +1386,15 @@ def _parse_contents_from_anthropic( self, content: Sequence[BetaContentBlock | BetaRawContentBlockDelta | BetaTextBlock], request_state: _AnthropicRequestState | None = None, + *, + streaming: bool = False, ) -> list[Content]: - """Parse contents from the Anthropic message.""" + """Parse contents from the Anthropic message. + + Anthropic does not supply Agent Framework merge modes. Streaming context selects the + type-specific mapping: text and function-call deltas append, while complete streamed + code-execution result snapshots replace. + """ request_state = request_state or _AnthropicRequestState() contents: list[Content] = [] for content_block in content: @@ -1380,6 +1403,7 @@ def _parse_contents_from_anthropic( contents.append( Content.from_text( text=content_block.text, + merge_mode="append" if streaming else None, raw_representation=content_block, annotations=self._parse_citations_from_anthropic(content_block), ) @@ -1407,6 +1431,7 @@ def _parse_contents_from_anthropic( raw_representation=content_block, ) ], + merge_mode="append" if streaming else None, raw_representation=content_block, ) ) @@ -1418,6 +1443,8 @@ def _parse_contents_from_anthropic( name=resolved_tool_name, arguments=content_block.input, informational_only=content_block.type == "server_tool_use", + hosted=content_block.type == "server_tool_use", + merge_mode="append" if streaming else None, raw_representation=content_block, ) ) @@ -1425,7 +1452,11 @@ def _parse_contents_from_anthropic( parsed_output: list[Content] | None = None if content_block.content: if isinstance(content_block.content, list): - parsed_output = self._parse_contents_from_anthropic(content_block.content, request_state) + parsed_output = self._parse_contents_from_anthropic( + content_block.content, + request_state, + streaming=streaming, + ) elif isinstance(content_block.content, (str, bytes)): parsed_output = [ Content.from_text( @@ -1434,7 +1465,11 @@ def _parse_contents_from_anthropic( ) ] else: - parsed_output = self._parse_contents_from_anthropic([content_block.content], request_state) + parsed_output = self._parse_contents_from_anthropic( + [content_block.content], + request_state, + streaming=streaming, + ) contents.append( Content.from_mcp_server_tool_result( call_id=content_block.tool_use_id, @@ -1500,6 +1535,7 @@ def _parse_contents_from_anthropic( call_id=content_block.tool_use_id, raw_representation=content_block, outputs=code_outputs, + merge_mode="replace" if streaming else None, ) ) case "bash_code_execution_tool_result": @@ -1656,6 +1692,7 @@ def _parse_contents_from_anthropic( call_id=call_id, name="", arguments=content_block.partial_json, + merge_mode="append" if streaming else None, raw_representation=content_block, ) ) @@ -1664,6 +1701,7 @@ def _parse_contents_from_anthropic( Content.from_text_reasoning( text=content_block.thinking, protected_data=getattr(content_block, "signature", None), + merge_mode="append" if streaming else None, raw_representation=content_block, ) ) @@ -1672,6 +1710,7 @@ def _parse_contents_from_anthropic( Content.from_text_reasoning( text=None, protected_data=content_block.signature, + merge_mode="append" if streaming else None, raw_representation=content_block, ) ) diff --git a/python/packages/anthropic/tests/test_anthropic_client.py b/python/packages/anthropic/tests/test_anthropic_client.py index c91a37765ed..3afb66eb5fb 100644 --- a/python/packages/anthropic/tests/test_anthropic_client.py +++ b/python/packages/anthropic/tests/test_anthropic_client.py @@ -1869,6 +1869,7 @@ def test_parse_contents_server_tool_use_input_json_delta_ignored( assert len(result) == 1 assert result[0].type == "function_call" assert result[0].informational_only is True + assert result[0].hosted is True assert request_state.active_call_content_type == "server_tool_use" # input_json_delta events after server_tool_use must be silently ignored @@ -1907,6 +1908,20 @@ def test_process_stream_event_simple(mock_anthropic_client: MagicMock) -> None: assert result is None +def test_streaming_content_uses_explicit_append_mode(mock_anthropic_client: MagicMock) -> None: + client = create_test_anthropic_client(mock_anthropic_client) + text_delta = MagicMock() + text_delta.type = "text_delta" + text_delta.text = "hello" + text_delta.citations = None + + contents = client._parse_contents_from_anthropic([text_delta], streaming=True) + + assert len(contents) == 1 + assert contents[0].type == "text" + assert contents[0].merge_mode == "append" + + async def test_inner_get_response(mock_anthropic_client: MagicMock) -> None: """Test _inner_get_response method.""" client = create_test_anthropic_client(mock_anthropic_client) @@ -2727,6 +2742,9 @@ def test_streaming_usage_not_double_counted(mock_anthropic_client: MagicMock) -> response = ChatResponse.from_updates(updates) + assert all( + content.merge_mode == "append" for update in updates for content in update.contents if content.type == "usage" + ) assert response.usage_details is not None assert response.usage_details["output_token_count"] == 25 assert response.usage_details["input_token_count"] == 10 diff --git a/python/packages/bedrock/agent_framework_bedrock/_chat_client.py b/python/packages/bedrock/agent_framework_bedrock/_chat_client.py index 302e5e30981..8e739418921 100644 --- a/python/packages/bedrock/agent_framework_bedrock/_chat_client.py +++ b/python/packages/bedrock/agent_framework_bedrock/_chat_client.py @@ -30,6 +30,7 @@ Message, ResponseStream, UsageDetails, + get_namespaced_value, validate_tool_mode, ) from agent_framework._settings import SecretString, load_settings @@ -62,7 +63,9 @@ # Marks text_reasoning content whose protected_data is base64 of a Bedrock ``redactedContent`` blob, # as opposed to the signature of a ``reasoningText`` block. -_REDACTED_REASONING_KEY = "bedrock.redacted_reasoning" +_BEDROCK_ADDITIONAL_PROPS_NAMESPACE = "bedrock" +_REDACTED_REASONING_KEY = "redacted_reasoning" +_LEGACY_REDACTED_REASONING_KEY = "bedrock.redacted_reasoning" @dataclass @@ -612,7 +615,11 @@ def _convert_content_to_bedrock_block(self, content: Content) -> dict[str, Any] return {"image": {"format": image_format, "source": {"bytes": _get_data_bytes(content)}}} case "text_reasoning": # Extended thinking with tool use requires reasoning blocks to be sent back unchanged. - if content.additional_properties.get(_REDACTED_REASONING_KEY): + if get_namespaced_value( + content.additional_properties, + _BEDROCK_ADDITIONAL_PROPS_NAMESPACE, + _REDACTED_REASONING_KEY, + ) or content.additional_properties.get(_LEGACY_REDACTED_REASONING_KEY): try: redacted = base64.b64decode(content.protected_data or "", validate=True) except ValueError: @@ -728,11 +735,21 @@ def _generate_reasoning_id() -> str: return f"reasoning-{uuid4().hex}" @staticmethod - def _redacted_reasoning_content(redacted: bytes, *, id: str, raw_representation: Any) -> Content: + def _redacted_reasoning_content( + redacted: bytes, + *, + id: str, + raw_representation: Any, + merge_mode: Literal["append", "replace"] | None = None, + ) -> Content: return Content.from_text_reasoning( id=id, protected_data=base64.b64encode(redacted).decode("utf-8"), - additional_properties={_REDACTED_REASONING_KEY: True}, + merge_mode=merge_mode, + additional_properties={ + _BEDROCK_ADDITIONAL_PROPS_NAMESPACE: {_REDACTED_REASONING_KEY: True}, + _LEGACY_REDACTED_REASONING_KEY: True, + }, raw_representation=raw_representation, ) @@ -774,18 +791,24 @@ def _process_converse_stream_event( if tool_use := block_start.get("start", {}).get("toolUse"): tool_call_ids[block_start.get("contentBlockIndex", 0)] = tool_use["toolUseId"] contents.append( - Content.from_function_call(call_id=tool_use["toolUseId"], name=tool_use["name"], arguments="") + Content.from_function_call( + call_id=tool_use["toolUseId"], + name=tool_use["name"], + arguments="", + merge_mode="append", + ) ) elif block_delta := event.get("contentBlockDelta"): delta = block_delta.get("delta", {}) if text_value := delta.get("text"): - contents.append(Content.from_text(text=text_value, raw_representation=delta)) + contents.append(Content.from_text(text=text_value, merge_mode="append", raw_representation=delta)) elif tool_use_delta := delta.get("toolUse"): contents.append( Content.from_function_call( call_id=tool_call_ids.get(block_delta.get("contentBlockIndex", 0), ""), name="", arguments=tool_use_delta.get("input", ""), + merge_mode="append", ) ) elif reasoning_delta := delta.get("reasoningContent"): @@ -799,7 +822,12 @@ def _process_converse_stream_event( # Merging keeps only the latest protected_data, so send everything received so far. block.redacted += bytes(redacted) contents.append( - self._redacted_reasoning_content(block.redacted, id=block.id, raw_representation=delta) + self._redacted_reasoning_content( + block.redacted, + id=block.id, + raw_representation=delta, + merge_mode="replace", + ) ) elif "text" in reasoning_delta or "signature" in reasoning_delta: contents.append( @@ -808,13 +836,14 @@ def _process_converse_stream_event( # "" rather than None, so a signed block without text is still replayed. text=reasoning_delta.get("text") or "", protected_data=reasoning_delta.get("signature"), + merge_mode="append", raw_representation=delta, ) ) elif message_stop := event.get("messageStop"): finish_reason = self._map_finish_reason(message_stop.get("stopReason")) elif (metadata := event.get("metadata")) and (usage_details := self._parse_usage(metadata.get("usage"))): - contents.append(Content.from_usage(usage_details=usage_details)) + contents.append(Content.from_usage(usage_details=usage_details, merge_mode="replace")) if not contents and finish_reason is None: return None return ChatResponseUpdate( diff --git a/python/packages/bedrock/tests/test_bedrock_client.py b/python/packages/bedrock/tests/test_bedrock_client.py index 209cbaa9315..80abecb56a6 100644 --- a/python/packages/bedrock/tests/test_bedrock_client.py +++ b/python/packages/bedrock/tests/test_bedrock_client.py @@ -2,6 +2,7 @@ from __future__ import annotations +import base64 import json import logging from collections import deque @@ -141,6 +142,10 @@ async def test_stream_yields_updates_as_converse_stream_events_arrive() -> None: assert [update.text for update in updates if update.text] == ["Checking", " the weather."] assert all(update.role == "assistant" for update in updates) + streamed_contents = [content for update in updates for content in update.contents] + assert all(content.merge_mode == "append" for content in streamed_contents if content.type == "text") + assert all(content.merge_mode == "append" for content in streamed_contents if content.type == "function_call") + assert all(content.merge_mode == "replace" for content in streamed_contents if content.type == "usage") assert response.text == "Checking the weather." function_call = next(content for content in response.messages[0].contents if content.type == "function_call") assert function_call.call_id == "call-1" @@ -311,12 +316,36 @@ def test_redacted_reasoning_content_is_replayed_unchanged() -> None: reasoning = response.messages[0].contents[0] assert reasoning.type == "text_reasoning" assert reasoning.text is None + assert reasoning.additional_properties["bedrock"]["redacted_reasoning"] is True + assert reasoning.additional_properties["bedrock.redacted_reasoning"] is True _, conversation = client._prepare_bedrock_messages(response.messages) assert conversation[0]["content"][0] == {"reasoningContent": {"redactedContent": redacted}} assert "toolUse" in conversation[0]["content"][1] +@pytest.mark.parametrize( + "additional_properties", + [ + {"bedrock": {"redacted_reasoning": True}}, + {"bedrock.redacted_reasoning": True}, + ], +) +def test_redacted_reasoning_replay_accepts_canonical_and_legacy_metadata( + additional_properties: dict[str, Any], +) -> None: + client = _make_client() + redacted = b"encrypted-reasoning" + content = Content.from_text_reasoning( + protected_data=base64.b64encode(redacted).decode("utf-8"), + additional_properties=additional_properties, + ) + + block = client._convert_content_to_bedrock_block(content) + + assert block == {"reasoningContent": {"redactedContent": redacted}} + + async def test_stream_redacted_reasoning_content_is_replayed_unchanged() -> None: stub = _StubBedrockStreamRuntime([ { diff --git a/python/packages/claude/agent_framework_claude/_agent.py b/python/packages/claude/agent_framework_claude/_agent.py index fe2983eefdc..5640dce24d9 100644 --- a/python/packages/claude/agent_framework_claude/_agent.py +++ b/python/packages/claude/agent_framework_claude/_agent.py @@ -942,7 +942,13 @@ async def _stream_run( if text: yield AgentResponseUpdate( role="assistant", - contents=[Content.from_text(text=text, raw_representation=message)], + contents=[ + Content.from_text( + text=text, + merge_mode="append", + raw_representation=message, + ) + ], raw_representation=message, ) elif delta_type == "thinking_delta": @@ -950,7 +956,13 @@ async def _stream_run( if thinking: yield AgentResponseUpdate( role="assistant", - contents=[Content.from_text_reasoning(text=thinking, raw_representation=message)], + contents=[ + Content.from_text_reasoning( + text=thinking, + merge_mode="append", + raw_representation=message, + ) + ], raw_representation=message, ) elif isinstance(message, AssistantMessage): @@ -1006,7 +1018,7 @@ async def _stream_run( ) if usage_details or finish_reason: yield AgentResponseUpdate( - contents=[Content.from_usage(usage_details, raw_representation=message)] + contents=[Content.from_usage(usage_details, merge_mode="replace", raw_representation=message)] if usage_details else None, finish_reason=cast(Any, finish_reason), diff --git a/python/packages/claude/tests/test_claude_agent.py b/python/packages/claude/tests/test_claude_agent.py index d8af59a0b92..b81326b0346 100644 --- a/python/packages/claude/tests/test_claude_agent.py +++ b/python/packages/claude/tests/test_claude_agent.py @@ -465,6 +465,8 @@ async def test_run_stream_yields_updates(self) -> None: assert updates[0].role == "assistant" assert updates[0].text == "Streaming " assert updates[1].text == "response" + assert updates[0].contents[0].merge_mode == "append" + assert updates[1].contents[0].merge_mode == "append" async def test_run_stream_final_response_captures_usage_and_finish_reason(self) -> None: """Test run(stream=True) final response includes ResultMessage metadata.""" @@ -500,10 +502,11 @@ async def test_run_stream_final_response_captures_usage_and_finish_reason(self) with patch("agent_framework_claude._agent.ClaudeSDKClient", return_value=mock_client): agent = ClaudeAgent() stream = agent.run("Hello", stream=True) - async for _ in stream: - pass + updates = [update async for update in stream] response = await stream.get_final_response() + usage_content = next(content for update in updates for content in update.contents if content.type == "usage") + assert usage_content.merge_mode == "replace" assert response.finish_reason == "length" assert response.usage_details == { "input_token_count": 7, diff --git a/python/packages/copilotstudio/agent_framework_copilotstudio/_agent.py b/python/packages/copilotstudio/agent_framework_copilotstudio/_agent.py index f276395f722..add38e39fb5 100644 --- a/python/packages/copilotstudio/agent_framework_copilotstudio/_agent.py +++ b/python/packages/copilotstudio/agent_framework_copilotstudio/_agent.py @@ -372,7 +372,12 @@ async def _process_activities(self, activities: AsyncIterable[Any], streaming: b ): yield Message( role="assistant", - contents=[Content.from_text(activity.text)], + contents=[ + Content.from_text( + activity.text, + merge_mode="append" if streaming else None, + ) + ], author_name=activity.from_property.name if activity.from_property else None, message_id=activity.id, raw_representation=activity, diff --git a/python/packages/copilotstudio/tests/test_copilot_agent.py b/python/packages/copilotstudio/tests/test_copilot_agent.py index 47dfe3a45bf..c8e2d5cb214 100644 --- a/python/packages/copilotstudio/tests/test_copilot_agent.py +++ b/python/packages/copilotstudio/tests/test_copilot_agent.py @@ -279,6 +279,7 @@ async def test_run_streaming_with_string_message(self, mock_copilot_client: Magi content = response.contents[0] assert content.type == "text" assert content.text == "Streaming response" + assert content.merge_mode == "append" response_count += 1 assert response_count == 1 diff --git a/python/packages/core/AGENTS.md b/python/packages/core/AGENTS.md index 6f6913b71d0..0bc85e26f19 100644 --- a/python/packages/core/AGENTS.md +++ b/python/packages/core/AGENTS.md @@ -76,6 +76,12 @@ agent_framework/ - **`ChatResponseUpdate`** - Streaming response update - **`AgentResponse`** / **`AgentResponseUpdate`** - Agent-level response wrappers - **`Content`** - Base class for message content (text, function calls, images, etc.) +- **Content identity and metadata** - `function_call.id` is Agent Framework occurrence identity; provider item IDs + belong in the provider namespace. `function_call.hosted` controls execution ownership, while `server_name` is the + descriptive hosted-server boundary and never implies `hosted=True`. Provider metadata uses one-level + `additional_properties` namespaces and streamed mergeable content uses `merge_mode`; response updates can use the + same field to replace all contents of one logical message; see + [spec 005](../../../docs/specs/005-python-content-identity-and-protocol-metadata.md). - **Computer use** - `ComputerSafetyCheck` and the `Content.from_computer_tool_call` / `from_computer_tool_result` constructors are experimental under `COMPUTER_USE`; the rest of `Content` retains its existing stage. Computer results can omit screenshots in core; OpenAI-based connectors require them when converting to Responses items. diff --git a/python/packages/core/agent_framework/__init__.py b/python/packages/core/agent_framework/__init__.py index 3953187578e..5a0b6800ab5 100644 --- a/python/packages/core/agent_framework/__init__.py +++ b/python/packages/core/agent_framework/__init__.py @@ -277,12 +277,15 @@ "UsageDetails", "add_usage_details", "detect_media_type_from_base64", + "get_namespaced_value", "map_chat_to_agent_update", + "merge_additional_properties", "merge_chat_options", "normalize_messages", "normalize_tools", "prepend_instructions_to_messages", "validate_chat_options", + "with_namespaced_value", "validate_tool_mode", "validate_tools", ), @@ -696,6 +699,7 @@ "executor", "function_middleware", "get_agent_mode", + "get_namespaced_value", "get_run_context", "handler", "included_messages", @@ -703,6 +707,7 @@ "keyword_check", "load_settings", "map_chat_to_agent_update", + "merge_additional_properties", "merge_chat_options", "normalize_function_invocation_configuration", "normalize_messages", @@ -727,6 +732,7 @@ "validate_tools", "validate_workflow_graph", "vectorstoremodel", + "with_namespaced_value", "workflow", ] diff --git a/python/packages/core/agent_framework/__init__.pyi b/python/packages/core/agent_framework/__init__.pyi index e8d1bc6e411..c5d5b0a250e 100644 --- a/python/packages/core/agent_framework/__init__.pyi +++ b/python/packages/core/agent_framework/__init__.pyi @@ -243,7 +243,9 @@ from ._types import ( UsageDetails, add_usage_details, detect_media_type_from_base64, + get_namespaced_value, map_chat_to_agent_update, + merge_additional_properties, merge_chat_options, normalize_messages, normalize_tools, @@ -251,6 +253,7 @@ from ._types import ( validate_chat_options, validate_tool_mode, validate_tools, + with_namespaced_value, ) from ._vector_filters import Filter, FilterGroup, FilterGroupOperator, FilterOperator, Param from ._vectors import ( @@ -651,6 +654,7 @@ __all__ = [ "executor", "function_middleware", "get_agent_mode", + "get_namespaced_value", "get_run_context", "handler", "included_messages", @@ -658,6 +662,7 @@ __all__ = [ "keyword_check", "load_settings", "map_chat_to_agent_update", + "merge_additional_properties", "merge_chat_options", "normalize_function_invocation_configuration", "normalize_messages", @@ -682,5 +687,6 @@ __all__ = [ "validate_tools", "validate_workflow_graph", "vectorstoremodel", + "with_namespaced_value", "workflow", ] diff --git a/python/packages/core/agent_framework/_compaction.py b/python/packages/core/agent_framework/_compaction.py index 4d5f967bfc8..6f4078eca35 100644 --- a/python/packages/core/agent_framework/_compaction.py +++ b/python/packages/core/agent_framework/_compaction.py @@ -1013,11 +1013,9 @@ def _serialize_content(content: Content) -> dict[str, Any]: else: payload.pop("protected_data", None) additional_properties = payload.get("additional_properties") - if isinstance(additional_properties, dict) and "encrypted_content" in additional_properties: - typed_properties = cast("dict[str, Any]", additional_properties) - payload["additional_properties"] = { - key: value for key, value in typed_properties.items() if key != "encrypted_content" - } + if isinstance(additional_properties, dict) and additional_properties: + filtered_properties = _strip_opaque_reasoning_payload(additional_properties) + payload["additional_properties"] = filtered_properties or {} return payload diff --git a/python/packages/core/agent_framework/_harness/_file_access.py b/python/packages/core/agent_framework/_harness/_file_access.py index f794fddfe78..0aa2d139b34 100644 --- a/python/packages/core/agent_framework/_harness/_file_access.py +++ b/python/packages/core/agent_framework/_harness/_file_access.py @@ -1961,15 +1961,14 @@ def _resolve_session_key(self, context: SessionContext) -> str: def _is_local_tool_call(function_call: Content) -> bool: """Return whether a function call targets this provider's local tools. - Hosted-tool calls carry a ``server_label`` in their - ``additional_properties`` and are a separate server-scoped approval + Hosted-tool calls set ``hosted=True`` and are a separate server-scoped approval boundary that must be passed through untouched (see :func:`agent_framework._tools._is_hosted_tool_approval`). These rules only ever auto-approve the provider's own local tools, so any call that - carries a ``server_label`` is rejected even if its name collides with a + is hosted is rejected even if its name collides with a file-access tool name. """ - return not function_call.additional_properties.get("server_label") + return not function_call.hosted @staticmethod def read_only_tools_auto_approval_rule(function_call: Content) -> bool: @@ -1983,7 +1982,7 @@ def read_only_tools_auto_approval_rule(function_call: Content) -> bool: the tools that modify it (``file_access_write``, ``file_access_delete``, ``file_access_replace``, and ``file_access_replace_lines``). - Hosted-tool calls (those carrying a ``server_label``) are never + Hosted-tool calls are never auto-approved, even when their name matches a file-access tool, so the rule stays scoped to this provider's local tools. @@ -2020,7 +2019,7 @@ def all_tools_auto_approval_rule(function_call: Content) -> bool: ``file_access_delete``, ``file_access_replace``, and ``file_access_replace_lines``). - Hosted-tool calls (those carrying a ``server_label``) are never + Hosted-tool calls are never auto-approved, even when their name matches a file-access tool, so the rule stays scoped to this provider's local tools. diff --git a/python/packages/core/agent_framework/_harness/_tool_approval.py b/python/packages/core/agent_framework/_harness/_tool_approval.py index 6e33b205149..e487a306a57 100644 --- a/python/packages/core/agent_framework/_harness/_tool_approval.py +++ b/python/packages/core/agent_framework/_harness/_tool_approval.py @@ -58,9 +58,12 @@ def _serialize_arguments(function_call: Content) -> dict[str, str]: def _server_label(function_call: Content) -> str | None: - """Return the hosted-tool server boundary for a function call, if present.""" - value = function_call.additional_properties.get("server_label") - return value if isinstance(value, str) else None + """Return the hosted-tool server boundary for a function call, if present. + + Legacy ``additional_properties["server_label"]`` is normalized into first-class + ``hosted`` and ``server_name`` by Content construction and restoration. + """ + return function_call.server_name if function_call.hosted else None def _content_from_state(value: Any) -> Content: diff --git a/python/packages/core/agent_framework/_sessions.py b/python/packages/core/agent_framework/_sessions.py index 73bfbaa603b..3b3b73d178f 100644 --- a/python/packages/core/agent_framework/_sessions.py +++ b/python/packages/core/agent_framework/_sessions.py @@ -872,7 +872,8 @@ def _approval_controls_to_keep(messages: Sequence[Message]) -> set[int]: content.id is not None and function_call is not None and function_call.call_id is not None - and not function_call.additional_properties.get("server_label") + # server_name is descriptive; only hosted controls execution ownership. + and not function_call.hosted and content.id not in unresolved_local_responses_by_id ): unresolved_local_responses_by_id[content.id] = content diff --git a/python/packages/core/agent_framework/_skills.py b/python/packages/core/agent_framework/_skills.py index 3bef5b39424..3e14844b85a 100644 --- a/python/packages/core/agent_framework/_skills.py +++ b/python/packages/core/agent_framework/_skills.py @@ -2200,15 +2200,14 @@ class SkillsProvider(ContextProvider): def _is_local_tool_call(function_call: Content) -> bool: """Return whether a function call targets this provider's local tools. - Hosted-tool calls carry a ``server_label`` in their - ``additional_properties`` and are a separate server-scoped approval + Hosted-tool calls set ``hosted=True`` and are a separate server-scoped approval boundary that must be passed through untouched (see :func:`agent_framework._tools._is_hosted_tool_approval`). These rules only ever auto-approve the provider's own local tools, so any call that - carries a ``server_label`` is rejected even if its name collides with a + is hosted is rejected even if its name collides with a skill tool name. """ - return not function_call.additional_properties.get("server_label") + return not function_call.hosted @staticmethod def read_only_tools_auto_approval_rule(function_call: Content) -> bool: @@ -2223,7 +2222,7 @@ def read_only_tools_auto_approval_rule(function_call: Content) -> bool: skill content (``load_skill`` and ``read_skill_resource``), while still prompting for script execution (``run_skill_script``). - Hosted-tool calls (those carrying a ``server_label``) are never + Hosted-tool calls are never auto-approved, even when their name matches a skill tool, so the rule stays scoped to this provider's local tools. @@ -2260,7 +2259,7 @@ def all_tools_auto_approval_rule(function_call: Content) -> bool: ``auto_approval_rules``) to automatically approve every skill tool, including the script execution tool (``run_skill_script``). - Hosted-tool calls (those carrying a ``server_label``) are never + Hosted-tool calls are never auto-approved, even when their name matches a skill tool, so the rule stays scoped to this provider's local tools. diff --git a/python/packages/core/agent_framework/_tools.py b/python/packages/core/agent_framework/_tools.py index eab23d5699f..6025b8b5281 100644 --- a/python/packages/core/agent_framework/_tools.py +++ b/python/packages/core/agent_framework/_tools.py @@ -2006,6 +2006,7 @@ def _function_execution_error_result( message = f"{message} Exception: {exception}" return _finalize_function_result( call_id=function_call.call_id, # type: ignore[arg-type] + id=function_call.id, result=message, exception=str(exception), base_additional_properties=function_call.additional_properties, @@ -2032,6 +2033,7 @@ def _function_argument_validation_error_result( message = f"{message} Exception: {exception_message}" return Content.from_function_result( call_id=function_call.call_id, # type: ignore[arg-type] + id=function_call.id, result=message, exception=exception_message, additional_properties=function_call.additional_properties, @@ -2109,13 +2111,14 @@ def _replacement_approval_request( def _finalize_function_result( *, call_id: str, + id: str | None = None, result: Any, base_additional_properties: Mapping[str, Any] | None = None, exception: str | None = None, context: FunctionInvocationContext | None = None, ) -> Content: """Build the stable function-result wrapper and apply private Host metadata.""" - from ._types import Content + from ._types import Content, merge_additional_properties carrier: _FunctionResultCarrier | None = None if context is not None: @@ -2127,10 +2130,14 @@ def _finalize_function_result( if carrier is not None: for key in carrier.exclusive_outer_keys: additional_properties.pop(key, None) - additional_properties.update(carrier.additional_properties) + additional_properties = merge_additional_properties( + additional_properties, + carrier.additional_properties, + ) function_result = Content.from_function_result( call_id=call_id, + id=id, result=result, exception=exception, additional_properties=additional_properties, @@ -2144,7 +2151,10 @@ def _finalize_function_result( updated_item.additional_properties = dict(updated_item.additional_properties) for key in carrier.exclusive_outer_keys | carrier.exclusive_item_keys: updated_item.additional_properties.pop(key, None) - updated_item.additional_properties.update(carrier.item_additional_properties) + updated_item.additional_properties = merge_additional_properties( + updated_item.additional_properties, + carrier.item_additional_properties, + ) updated_items[index] = updated_item function_result.items = updated_items return function_result @@ -2283,6 +2293,7 @@ async def _auto_invoke_function( ) return _finalize_function_result( call_id=function_call_content.call_id, # type: ignore[arg-type] + id=function_call_content.id, result=function_result, base_additional_properties=function_call_content.additional_properties, context=direct_context, @@ -2383,6 +2394,7 @@ def ensure_short_circuit_arguments_are_authorized() -> None: return _finalize_function_result( call_id=call_id, + id=function_call_content.id, result=function_result, base_additional_properties=function_call_content.additional_properties, context=middleware_context, @@ -2409,6 +2421,7 @@ def ensure_short_circuit_arguments_are_authorized() -> None: # Store result in exception for caller to extract term_exc.result = _finalize_function_result( call_id=call_id, + id=function_call_content.id, result=middleware_context.result, base_additional_properties=function_call_content.additional_properties, context=middleware_context, @@ -2441,7 +2454,11 @@ def _get_tool_map( def _is_actionable_function_call(content: Content) -> bool: - return content.type == "function_call" and not content.informational_only + """Return whether core may execute the call locally. + + ``server_name`` is descriptive and intentionally does not imply hosted ownership. + """ + return content.type == "function_call" and not content.informational_only and not content.hosted def _underlying_function_call(content: Content) -> Content: @@ -2506,6 +2523,7 @@ async def _execute_single_function_call( return [ Content.from_function_result( call_id=source_function_call.call_id, # type: ignore[arg-type] + id=source_function_call.id, result=exc.result, ) ], True @@ -2516,12 +2534,13 @@ async def _execute_single_function_call( for item in propagated_contents: item.call_id = call_id if not item.id: - item.id = call_id + item.id = source_function_call.id or call_id if propagated_contents: return propagated_contents, False return [ Content.from_function_result( call_id=call_id, # type: ignore[arg-type] + id=source_function_call.id, result="Tool requires user input but no request details were provided.", exception="UserInputRequiredException", ) @@ -2861,14 +2880,11 @@ def _clear_internal_conversation_id(response: ChatResponse[Any]) -> ChatResponse def _is_hosted_tool_approval(content: Any) -> bool: """Check if a function_approval_request/response is for a hosted tool (e.g. MCP). - Hosted tool approvals have a server_label in function_call.additional_properties - and should be passed through to the API untouched rather than processed locally. + Hosted tool approvals are provider-executed and should be passed through to + the API untouched rather than processed locally. """ fc = getattr(content, "function_call", None) - if fc is None: - return False - ap = getattr(fc, "additional_properties", None) - return bool(ap and ap.get("server_label")) + return bool(fc is not None and getattr(fc, "hosted", False)) def _is_approval_granted(value: Any) -> bool: @@ -4534,6 +4550,7 @@ def find_approval_occurrence(approval_id: str) -> _ApprovalCallOccurrence | None replacements = [ Content.from_function_result( call_id=call_id, + id=occurrence.function_call.id if occurrence is not None else content.function_call.id, result="Error: Tool call invocation was rejected by user.", additional_properties=content.function_call.additional_properties, ) @@ -5089,6 +5106,7 @@ async def _resolve_deferred_mixed_pause_calls( [ Content.from_function_result( call_id=function_call.call_id, # type: ignore[arg-type] + id=function_call.id, result="Error: Function invocation limit reached before the deferred tool could run.", exception="FunctionInvocationLimit", additional_properties=function_call.additional_properties, @@ -5614,6 +5632,7 @@ async def _settle_dangling_service_function_calls( error_results.append( Content.from_function_result( call_id=underlying_call.call_id, + id=underlying_call.id, result=error_result, exception=error_exception, additional_properties=underlying_call.additional_properties, @@ -5903,6 +5922,7 @@ async def _stream_response_with_function_invocation( ) -> AsyncIterable[ChatResponseUpdate]: """Run the streaming function invocation loop.""" from ._middleware import MiddlewareFailure + from ._types import ChatResponseUpdate, Content, add_usage_details errors_in_a_row = 0 total_function_calls = int(budget_state.get("total_function_calls", 0) or 0) @@ -5910,9 +5930,65 @@ async def _stream_response_with_function_invocation( max_duration_seconds = self.function_invocation_configuration.get("max_duration_seconds") prepared_messages = _copy_messages_for_function_invocation(messages) response: ChatResponse[Any] | None = None + completed_usage: UsageDetails | None = None + turn_usage: UsageDetails | None = None + last_projected_usage: UsageDetails | None = None max_iterations = self.function_invocation_configuration.get("max_iterations", DEFAULT_MAX_ITERATIONS) attempt_start = int(budget_state.get("attempt_count", 0) or 0) + def project_cumulative_usage(update: ChatResponseUpdate) -> ChatResponseUpdate: + """Project one provider turn's usage as a cumulative outer-stream snapshot.""" + nonlocal turn_usage, last_projected_usage + projected_contents: list[Content] | None = None + for index, content in enumerate(update.contents): + if content.type != "usage": + continue + if content.merge_mode == "replace": + turn_usage = copy.deepcopy(content.usage_details) + else: + turn_usage = add_usage_details(turn_usage, content.usage_details) + if completed_usage is None: + continue + if projected_contents is None: + projected_contents = list(update.contents) + cumulative_usage = add_usage_details(completed_usage, turn_usage) + # Copy only the objects mutated below. Other fields stay read-only, and raw provider objects may not + # support deepcopy. + projected_content = copy.copy(content) + projected_content.usage_details = copy.deepcopy(cumulative_usage) + projected_content.merge_mode = "replace" + projected_contents[index] = projected_content + last_projected_usage = cumulative_usage + if projected_contents is None: + return update + projected_update = copy.copy(update) + projected_update.contents = projected_contents + return projected_update + + def finish_usage_turn(final_usage: UsageDetails | None) -> ChatResponseUpdate | None: + """Complete one provider turn and emit a correction if its final usage was not streamed.""" + nonlocal completed_usage, turn_usage, last_projected_usage + turn_total = final_usage if final_usage is not None else turn_usage + if turn_total is None: + turn_usage = None + last_projected_usage = None + return None + had_completed_usage = completed_usage is not None + completed_usage = add_usage_details(completed_usage, turn_total) + correction = None + if had_completed_usage and completed_usage != last_projected_usage: + correction = ChatResponseUpdate( + contents=[ + Content.from_usage( + copy.deepcopy(completed_usage), + merge_mode="replace", + ) + ] + ) + turn_usage = None + last_projected_usage = None + return correction + async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> None: await self._settle_dangling_service_function_calls( super_get_response=super_get_response, @@ -6050,9 +6126,11 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non update = _drop_unexecutable_tool_contents_from_update(update) if update is None: continue - yield update + yield project_cumulative_usage(update) response = await _await_provider_call(inner_stream.get_final_response, on_invalidated) + if usage_correction := finish_usage_turn(response.usage_details): + yield usage_correction fallback_added = False if options.get("tool_choice") == "none" and budget_state.get("truncated"): fallback_added = _ensure_function_invocation_limit_fallback_response(response) @@ -6148,13 +6226,17 @@ async def settle_approval_replay_calls(function_calls: Sequence[Content]) -> Non client_kwargs=request_kwargs, ), ) + turn_usage = None + last_projected_usage = None async with final_inner_stream: async for update in _iterate_provider_stream(final_inner_stream, on_invalidated): update = _drop_unexecutable_tool_contents_from_update(update) if update is None: continue - yield update + yield project_cumulative_usage(update) final_response = await _await_provider_call(final_inner_stream.get_final_response, on_invalidated) + if usage_correction := finish_usage_turn(final_response.usage_details): + yield usage_correction fallback_added = _ensure_function_invocation_limit_fallback_response(final_response) self._update_function_invocation_continuation_state( request_kwargs, diff --git a/python/packages/core/agent_framework/_types.py b/python/packages/core/agent_framework/_types.py index 6ffebae8700..c08fec0ba70 100644 --- a/python/packages/core/agent_framework/_types.py +++ b/python/packages/core/agent_framework/_types.py @@ -433,6 +433,14 @@ def _restore_compaction_annotation_in_additional_properties( "function_approval_response", "oauth_consent_request", ] +_MERGEABLE_CONTENT_TYPES: Final[frozenset[str]] = frozenset({ + "text", + "text_reasoning", + "function_call", + "usage", + "code_interpreter_tool_call", + "code_interpreter_tool_result", +}) class TextSpanRegion(TypedDict, total=False): @@ -469,6 +477,7 @@ class ComputerSafetyCheck(TypedDict, total=False): ContentT = TypeVar("ContentT", bound="Content") _MODEL_OUTPUT_KIND_KEY = "model_output_kind" _MODEL_OUTPUT_REFUSAL = "refusal" +_REASONING_TEXT_KEY = "reasoning_text" # endregion @@ -497,6 +506,83 @@ class UsageDetails(TypedDict, total=False, extra_items=int): reasoning_output_token_count: int | None +def get_namespaced_value( + additional_properties: Mapping[str, Any] | None, + namespace: str, + key: str, + default: Any = None, +) -> Any: + """Read one value from a namespaced additional-properties mapping.""" + if not additional_properties: + return default + namespace_values = additional_properties.get(namespace) + if not isinstance(namespace_values, Mapping): + return default + return cast("Mapping[str, Any]", namespace_values).get(key, default) + + +def with_namespaced_value( + additional_properties: Mapping[str, Any] | None, + namespace: str, + key: str, + value: Any, +) -> dict[str, Any]: + """Return a copy with one namespaced additional-properties value updated.""" + updated = dict(additional_properties or {}) + namespace_values = updated.get(namespace) + updated_namespace = ( + dict(cast("Mapping[str, Any]", namespace_values)) if isinstance(namespace_values, Mapping) else {} + ) + updated_namespace[key] = value + updated[namespace] = updated_namespace + return updated + + +def merge_additional_properties( + existing: Mapping[str, Any] | None, + incoming: Mapping[str, Any] | None, + *, + incoming_wins: bool = True, +) -> dict[str, Any]: + """Merge additional properties one level deep while preserving the site's conflict rule.""" + first, second = (existing, incoming) if incoming_wins else (incoming, existing) + merged = _copy_additional_properties(first) + for key, value in (second or {}).items(): + current = merged.get(key) + if isinstance(current, Mapping) and isinstance(value, Mapping): + merged[key] = {**current, **value} + else: + merged[key] = dict(cast("Mapping[str, Any]", value)) if isinstance(value, Mapping) else value + return merged + + +def _copy_additional_properties(additional_properties: Mapping[str, Any] | None) -> dict[str, Any]: + """Copy an additional-properties mapping and its top-level mapping values.""" + copied = dict(additional_properties or {}) + for key, value in copied.items(): + if isinstance(value, Mapping): + copied[key] = dict(cast("Mapping[str, Any]", value)) + return copied + + +def _model_output_kind(additional_properties: Mapping[str, Any]) -> Any: + """Read the canonical model-output kind with legacy flat-key compatibility.""" + return get_namespaced_value( + additional_properties, + "core", + _MODEL_OUTPUT_KIND_KEY, + additional_properties.get(_MODEL_OUTPUT_KIND_KEY), + ) + + +def _has_reasoning_text_marker(additional_properties: Mapping[str, Any]) -> bool: + """Check the canonical OpenAI reasoning-text marker and its legacy location.""" + if _REASONING_TEXT_KEY in additional_properties: + return True + openai_properties = additional_properties.get("openai") + return isinstance(openai_properties, Mapping) and _REASONING_TEXT_KEY in openai_properties + + def add_usage_details(usage1: UsageDetails | None, usage2: UsageDetails | None) -> UsageDetails: """Add two UsageDetails dictionaries by summing all numeric values. @@ -610,6 +696,7 @@ def __init__( tool_name: str | None = None, server_name: str | None = None, output: Any = None, + hosted: bool | None = None, # Function approval fields id: str | None = None, function_call: Content | None = None, @@ -618,6 +705,7 @@ def __init__( # OAuth consent fields consent_link: str | None = None, # Common fields + merge_mode: Literal["append", "replace"] | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any | None = None, @@ -626,12 +714,40 @@ def __init__( Prefer using the classmethod constructors like `Content.from_text()` instead of calling __init__ directly. """ + if merge_mode not in (None, "append", "replace"): + raise ValueError(f"Invalid merge mode: {merge_mode!r}") + if merge_mode is not None and type not in _MERGEABLE_CONTENT_TYPES: + raise ValueError(f"Content type {type!r} does not support merge_mode.") + + restored_properties = _restore_compaction_annotation_in_additional_properties(additional_properties) + copied_properties = _copy_additional_properties(restored_properties) + legacy_server_label = copied_properties.get("server_label") + if legacy_server_label is not None and not isinstance(legacy_server_label, str): + raise ValueError("Legacy function-call server_label must be a string or None.") + if type == "function_call": + if hosted is not None and not isinstance(hosted, bool): + raise ValueError("Function-call hosted must be true or false.") + if hosted is None: + hosted = bool(legacy_server_label) + if hosted: + warnings.warn( + "Function-call content inferred hosted=True from legacy additional_properties['server_label']. " + "Set hosted explicitly; the legacy inference will be removed in a future release.", + FutureWarning, + stacklevel=2, # Report the constructor caller rather than this migration shim. + ) + if server_name is None and legacy_server_label: + server_name = legacy_server_label + elif server_name is not None and legacy_server_label is not None and server_name != legacy_server_label: + raise ValueError("Function-call server_name conflicts with legacy server_label.") + elif hosted is not None: + raise ValueError("hosted is only valid for function_call content.") + self.type = type self.annotations = annotations - self.additional_properties: dict[str, Any] = ( - _restore_compaction_annotation_in_additional_properties(additional_properties) or {} - ) + self.additional_properties = copied_properties self.raw_representation = raw_representation + self.merge_mode: Literal["append", "replace"] | None = merge_mode # Set all content-specific attributes. A text content always carries a string, so # concatenation and `Message.text` never see None. @@ -670,6 +786,7 @@ def __init__( self.tool_name = tool_name self.server_name = server_name self.output = output + self.hosted = hosted self.id = id self.function_call = function_call self.user_input_request = user_input_request @@ -716,12 +833,39 @@ def __getstate__(self) -> dict[str, Any]: def __setstate__(self, state: dict[str, Any] | tuple[dict[str, Any], dict[str, Any]]) -> None: """Restore pickle state and reset runtime-only shallow-copy fields.""" restore_pickle_state(self, state, self._PICKLE_OMIT_FIELDS) + if not hasattr(self, "merge_mode"): + self.merge_mode = None + if self.type != "function_call": + if not hasattr(self, "hosted"): + self.hosted = None + return + legacy_server_label = self.additional_properties.get("server_label") + if legacy_server_label is not None and not isinstance(legacy_server_label, str): + raise ValueError("Legacy function-call server_label must be a string or None.") + if not hasattr(self, "hosted"): + self.hosted = bool(legacy_server_label) + if self.hosted: + warnings.warn( + "Function-call content inferred hosted=True from legacy additional_properties['server_label']. " + "Set hosted explicitly; the legacy inference will be removed in a future release.", + FutureWarning, + stacklevel=2, # Report the unpickle caller rather than this migration shim. + ) + elif type(self.hosted) is not bool: + raise ValueError("Function-call hosted must be true or false.") + if self.server_name is None and legacy_server_label: + self.server_name = legacy_server_label + elif ( + self.server_name is not None and legacy_server_label is not None and self.server_name != legacy_server_label + ): + raise ValueError("Function-call server_name conflicts with legacy server_label.") @classmethod def from_text( cls: type[ContentT], text: str, *, + merge_mode: Literal["append", "replace"] | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -730,6 +874,7 @@ def from_text( return cls( "text", text=text, + merge_mode=merge_mode, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -742,6 +887,8 @@ def from_text_reasoning( id: str | None = None, text: str | None = None, protected_data: str | None = None, + status: str | None = None, + merge_mode: Literal["append", "replace"] | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -752,6 +899,8 @@ def from_text_reasoning( id=id, text=text, protected_data=protected_data, + status=status, + merge_mode=merge_mode, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -763,6 +912,8 @@ def from_data( data: bytes, media_type: str, *, + name: str | None = None, + file_id: str | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -790,6 +941,8 @@ def from_data( content = Content.from_data(data=data_bytes, media_type=media_type) Keyword Args: + name: Optional filename or display name for the data. + file_id: Optional provider file identifier associated with the data. annotations: Optional annotations associated with the content. additional_properties: Optional additional properties. raw_representation: Optional raw representation from an underlying implementation. @@ -828,6 +981,8 @@ def from_data( "data", uri=f"data:{media_type};base64,{encoded_data}", media_type=media_type, + name=name, + file_id=file_id, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -839,6 +994,8 @@ def from_uri( uri: str, *, media_type: str | None = None, + name: str | None = None, + file_id: str | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -857,6 +1014,8 @@ def from_uri( Keyword Args: media_type: The MIME type of the data (e.g., "image/png", "application/pdf"). This is optional but recommended for external URIs. + name: Optional filename or display name for the resource. + file_id: Optional provider file identifier associated with the resource. annotations: Optional annotations associated with the content. additional_properties: Optional additional properties. raw_representation: Optional raw representation from an underlying implementation. @@ -890,6 +1049,8 @@ def from_uri( """ return cls( **_validate_uri(uri, media_type), + name=name, + file_id=file_id, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -927,6 +1088,10 @@ def from_function_call( exception: str | None = None, informational_only: bool = False, id: str | None = None, + hosted: bool | None = None, + server_name: str | None = None, + status: str | None = None, + merge_mode: Literal["append", "replace"] | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -947,6 +1112,11 @@ def from_function_call( executed by Agent Framework function invocation. id: Stable Agent Framework identity for this occurrence. When omitted, the function invocation layer assigns one before a locally actionable call is processed. + hosted: Whether the provider or service, rather than Agent Framework, executes this call. + server_name: Descriptive server boundary. Supplying it does not mark the call as hosted; set + ``hosted=True`` separately when the provider or service executes the call. + status: Provider-reported function-call status. + merge_mode: How this streaming update combines with an earlier update for the same occurrence. annotations: Optional annotations attached to this content item. additional_properties: Extra provider-specific properties to preserve with the content item. raw_representation: The original provider-specific object or payload this content item was created from. @@ -962,6 +1132,10 @@ def from_function_call( exception=exception, informational_only=informational_only, id=id, + hosted=hosted, + server_name=server_name, + status=status, + merge_mode=merge_mode, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -972,8 +1146,10 @@ def from_function_result( cls: type[ContentT], call_id: str, *, + id: str | None = None, result: Any = None, exception: str | None = None, + status: str | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -991,9 +1167,11 @@ def from_function_result( result: The tool output. Accepts a ``list[Content]`` (the canonical form produced by :meth:`~FunctionTool.parse_result`), a plain ``str``, or any other value (which is stringified). + id: Stable Agent Framework identity of the function-call occurrence this result settles. exception: Host-internal diagnostic information when the function call failed. Treat it as potentially sensitive regardless of whether it came from a tool, middleware, provider, or caller. Serialization replaces it with a fixed failure marker; use ``result`` for channel-visible error text. + status: Provider-reported function-result status. annotations: Optional annotations for the content. additional_properties: Optional additional properties. raw_representation: Optional raw representation from the provider. @@ -1019,10 +1197,12 @@ def from_function_result( return cls( "function_result", + id=id, call_id=call_id, result=text_result, items=items_list, exception=exception, + status=status, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -1083,6 +1263,7 @@ def from_usage( cls: type[ContentT], usage_details: UsageDetails, *, + merge_mode: Literal["append", "replace"] | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -1091,6 +1272,7 @@ def from_usage( return cls( "usage", usage_details=usage_details, + merge_mode=merge_mode, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -1142,6 +1324,8 @@ def from_code_interpreter_tool_call( *, call_id: str | None = None, inputs: Sequence[Content] | None = None, + status: str | None = None, + merge_mode: Literal["append", "replace"] | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -1151,6 +1335,8 @@ def from_code_interpreter_tool_call( "code_interpreter_tool_call", call_id=call_id, inputs=list(inputs) if inputs is not None else None, + status=status, + merge_mode=merge_mode, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -1162,6 +1348,8 @@ def from_code_interpreter_tool_result( *, call_id: str | None = None, outputs: Sequence[Content] | None = None, + status: str | None = None, + merge_mode: Literal["append", "replace"] | None = None, annotations: Sequence[Annotation] | None = None, additional_properties: MutableMapping[str, Any] | None = None, raw_representation: Any = None, @@ -1171,6 +1359,8 @@ def from_code_interpreter_tool_result( "code_interpreter_tool_result", call_id=call_id, outputs=list(outputs) if outputs is not None else None, + status=status, + merge_mode=merge_mode, annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -1477,7 +1667,7 @@ def from_function_approval_request( function_call.type == "function_call" and function_call.id is not None and id != function_call.id - and function_call.additional_properties.get("server_label") is None + and not function_call.hosted and not (additional_properties or {}).get("_replacement_approval_request", False) ): warnings.warn( @@ -1620,11 +1810,13 @@ def _to_dict( "tool_name", "server_name", "output", + "hosted", "function_call", "user_input_request", "approved", "id", "consent_link", + "merge_mode", "additional_properties", ) @@ -1679,6 +1871,8 @@ def from_dict(cls: type[ContentT], data: Mapping[str, Any]) -> ContentT: annotations = remaining.pop("annotations", None) additional_properties = remaining.pop("additional_properties", None) raw_representation = remaining.pop("raw_representation", None) + if content_type == "function_call" and "hosted" in remaining and remaining["hosted"] is None: + raise ValueError("Function-call hosted must be true or false.") # Special handling for DataContent with data and media_type if content_type == "data" and "data" in remaining and "media_type" in remaining: @@ -1689,6 +1883,8 @@ def from_dict(cls: type[ContentT], data: Mapping[str, Any]) -> ContentT: return cls.from_data( remaining["data"], remaining["media_type"], + name=remaining.get("name"), + file_id=remaining.get("file_id"), annotations=annotations, additional_properties=additional_properties, raw_representation=raw_representation, @@ -1735,6 +1931,10 @@ def __add__(self, other: Content) -> Content: return self._add_function_call_content(other) if self.type == "usage": return self._add_usage_content(other) + if self.type in ("code_interpreter_tool_call", "code_interpreter_tool_result"): + merged = deepcopy(self) + _merge_code_interpreter_content(merged, other) + return merged raise ContentError(f"Addition not supported for content type: {self.type}") def _add_text_content(self, other: Content) -> Content: @@ -1745,7 +1945,8 @@ def _add_text_content(self, other: Content) -> Content: """ return Content( "text", - text=self.text + other.text, # type: ignore[attr-defined, operator] + text=other.text if other.merge_mode == "replace" else self.text + other.text, # type: ignore[attr-defined, operator] + merge_mode=other.merge_mode, annotations=_combine_annotations(self.annotations, other.annotations), additional_properties=_combine_additional_props(self.additional_properties, other.additional_properties), raw_representation=_combine_raw_representations(self.raw_representation, other.raw_representation), @@ -1775,19 +1976,31 @@ def _add_text_reasoning_content(self, other: Content) -> Content: if ( self_text and other_text - and ("reasoning_text" in self.additional_properties) != ("reasoning_text" in other.additional_properties) + and _has_reasoning_text_marker(self.additional_properties) + != _has_reasoning_text_marker(other.additional_properties) ): raise AdditionItemMismatch("Cannot merge reasoning text with a reasoning summary") - combined_text = None if self.text is None and other.text is None else self_text + other_text + if other.merge_mode == "replace": + combined_text = other.text + elif self.text is None and other.text is None: + combined_text = None + else: + combined_text = self_text + other_text # Handle protected_data replacement - protected_data = other.protected_data if other.protected_data is not None else self.protected_data + protected_data = ( + other.protected_data + if other.merge_mode == "replace" or other.protected_data is not None + else self.protected_data + ) return Content( "text_reasoning", id=combined_id, text=combined_text, protected_data=protected_data, + status=other.status if other.status is not None else self.status, + merge_mode=other.merge_mode, annotations=_combine_annotations(self.annotations, other.annotations), additional_properties=_combine_additional_props(self.additional_properties, other.additional_properties), raw_representation=_combine_raw_representations(self.raw_representation, other.raw_representation), @@ -1805,7 +2018,9 @@ def _add_function_call_content(self, other: Content) -> Content: self_arguments = getattr(self, "arguments", None) other_arguments = getattr(other, "arguments", None) - if not self_arguments: + if other.merge_mode == "replace": + arguments = other_arguments + elif not self_arguments: arguments: str | Mapping[str, Any] | None = other_arguments elif not other_arguments: arguments = self_arguments @@ -1816,16 +2031,28 @@ def _add_function_call_content(self, other: Content) -> Content: else: raise TypeError("Incompatible argument types") + self_server_name = getattr(self, "server_name", None) + other_server_name = getattr(other, "server_name", None) + if self_server_name and other_server_name and self_server_name != other_server_name: + raise AdditionItemMismatch("Cannot merge function calls from different servers") + + self_name = getattr(self, "name", None) + other_name = getattr(other, "name", None) return Content( "function_call", call_id=self_call_id or other_call_id, - name=getattr(self, "name", None) or getattr(other, "name", None), + name=other_name if other.merge_mode == "replace" else self_name or other_name, arguments=arguments, id=self.id or other.id, + hosted=bool(self.hosted or other.hosted), + server_name=self_server_name or other_server_name, + status=other.status if other.status is not None else self.status, + merge_mode=other.merge_mode, user_input_request=self.user_input_request or other.user_input_request, - exception=getattr(self, "exception", None) or getattr(other, "exception", None), + exception=other.exception if other.exception is not None else self.exception, informational_only=getattr(self, "informational_only", False) or getattr(other, "informational_only", False), + annotations=_combine_annotations(self.annotations, other.annotations), additional_properties=_combine_additional_props(self.additional_properties, other.additional_properties), raw_representation=_combine_raw_representations(self.raw_representation, other.raw_representation), ) @@ -1834,7 +2061,13 @@ def _add_usage_content(self, other: Content) -> Content: """Add two UsageContent instances by combining their usage details.""" return Content( "usage", - usage_details=add_usage_details(self.usage_details, other.usage_details), + usage_details=( + deepcopy(other.usage_details) + if other.merge_mode == "replace" + else add_usage_details(self.usage_details, other.usage_details) + ), + merge_mode=other.merge_mode, + annotations=_combine_annotations(self.annotations, other.annotations), additional_properties=_combine_additional_props(self.additional_properties, other.additional_properties), raw_representation=_combine_raw_representations(self.raw_representation, other.raw_representation), ) @@ -1917,10 +2150,11 @@ def _combine_additional_props( self_additional_properties: dict[str, Any], other_additional_properties: dict[str, Any] ) -> dict[str, Any]: """Combine additional properties for addition operations.""" - return { - **other_additional_properties, - **self_additional_properties, - } + return merge_additional_properties( + self_additional_properties, + other_additional_properties, + incoming_wins=False, + ) def _combine_raw_representations( @@ -2087,8 +2321,8 @@ def __init__( self.contents = parsed_contents self.author_name = author_name self.message_id = message_id - self.additional_properties = ( - _restore_compaction_annotation_in_additional_properties(additional_properties) or {} + self.additional_properties = _copy_additional_properties( + _restore_compaction_annotation_in_additional_properties(additional_properties) ) self.raw_representation = raw_representation @@ -2233,6 +2467,8 @@ def _process_update(response: ChatResponse | AgentResponse, update: ChatResponse response.messages.append(message) else: message = response.messages[-1] + if getattr(update, "merge_mode", None) == "replace": + message.contents.clear() # Incorporate the update's properties into the message. if update.author_name is not None: message.author_name = update.author_name @@ -2255,10 +2491,14 @@ def _process_update(response: ChatResponse | AgentResponse, update: ChatResponse case "function_call": _merge_function_call_content(message, content) case "usage": - if response.usage_details is None: + if content.merge_mode == "replace": + response.usage_details = deepcopy(content.usage_details) + elif response.usage_details is None: response.usage_details = UsageDetails() - # mypy doesn't narrow type based on match/case, but we know this is UsageContent - response.usage_details = add_usage_details(response.usage_details, content.usage_details) + response.usage_details = add_usage_details(response.usage_details, content.usage_details) + else: + # mypy doesn't narrow type based on match/case, but we know this is UsageContent + response.usage_details = add_usage_details(response.usage_details, content.usage_details) case _: message.contents.append(content) # Incorporate the update's properties into the response. @@ -2267,7 +2507,11 @@ def _process_update(response: ChatResponse | AgentResponse, update: ChatResponse if update.created_at is not None: response.created_at = update.created_at if update.additional_properties is not None: - response.additional_properties.update(update.additional_properties) + response.additional_properties = merge_additional_properties( + response.additional_properties, + update.additional_properties, + incoming_wins=True, + ) if response.raw_representation is None: response.raw_representation = [] if not isinstance(response.raw_representation, list): @@ -2360,9 +2604,11 @@ def _apply_response_tail_to_update( update.continuation_token = response.continuation_token _copy_operation_state(response, update) if response.additional_properties: - merged = dict(update.additional_properties) if update.additional_properties else {} - merged.update(response.additional_properties) - update.additional_properties = merged + update.additional_properties = merge_additional_properties( + update.additional_properties, + response.additional_properties, + incoming_wins=True, + ) if response.usage_details is not None: update.contents.append(Content.from_usage(response.usage_details)) @@ -2421,10 +2667,16 @@ def _merge_content_run(run: list[Content], type_str: Literal["text", "text_reaso the aggregate. """ head = run[0] - merged_props: dict[str, Any] = {} - for content in reversed(run[1:]): - merged_props.update(content.additional_properties) - merged_props.update(deepcopy(head.additional_properties)) + merged_props: dict[str, Any] = deepcopy(head.additional_properties) + for content in run[1:]: + for key, value in content.additional_properties.items(): + current = merged_props.get(key) + if isinstance(current, dict) and isinstance(value, Mapping): + current_mapping = cast("dict[str, Any]", current) + for nested_key, nested_value in cast("Mapping[str, Any]", value).items(): + current_mapping.setdefault(nested_key, nested_value) + elif key not in merged_props: + merged_props[key] = dict(cast("Mapping[str, Any]", value)) if isinstance(value, Mapping) else value annotation_seqs = [ deepcopy(c.annotations) if c is head else c.annotations for c in run if c.annotations is not None ] @@ -2440,18 +2692,35 @@ def _merge_content_run(run: list[Content], type_str: Literal["text", "text_reaso for raw in non_null_raws: raw_representation.extend(raw if isinstance(raw, list) else [raw]) if type_str == "text": + text = "" + for content in run: + text = (content.text or "") if content.merge_mode == "replace" else text + (content.text or "") return Content( "text", - text="".join(c.text or "" for c in run), + text=text, + merge_mode=run[-1].merge_mode, annotations=annotations, additional_properties=merged_props, raw_representation=raw_representation, ) + text: str | None = None + protected_data: str | None = None + for content in run: + if content.merge_mode == "replace": + text = content.text + protected_data = content.protected_data + continue + if content.text is not None: + text = (text or "") + content.text + if content.protected_data is not None: + protected_data = content.protected_data return Content( "text_reasoning", id=next((c.id for c in run if c.id), run[-1].id), - text=None if all(c.text is None for c in run) else "".join(c.text or "" for c in run), - protected_data=next((c.protected_data for c in reversed(run) if c.protected_data is not None), None), + text=text, + protected_data=protected_data, + status=next((c.status for c in reversed(run) if c.status is not None), None), + merge_mode=run[-1].merge_mode, annotations=annotations, additional_properties=merged_props, raw_representation=raw_representation, @@ -2489,8 +2758,8 @@ def flush() -> None: if type_str == "text": # A folded run keeps the first chunk's value for this key, so the # run head alone decides whether the next chunk joins or splits. - if run[0].additional_properties.get(_MODEL_OUTPUT_KIND_KEY) != content.additional_properties.get( - _MODEL_OUTPUT_KIND_KEY + if _model_output_kind(run[0].additional_properties) != _model_output_kind( + content.additional_properties ): flush() else: @@ -2498,14 +2767,14 @@ def flush() -> None: if (run_id and other_id and run_id != other_id) or ( run_has_text and getattr(content, "text", None) - and (run_has_reasoning_text != ("reasoning_text" in content.additional_properties)) + and (run_has_reasoning_text != _has_reasoning_text_marker(content.additional_properties)) ): flush() run.append(content) if type_str == "text_reasoning": run_id = run_id or getattr(content, "id", None) run_has_text = run_has_text or bool(getattr(content, "text", None)) - run_has_reasoning_text = run_has_reasoning_text or "reasoning_text" in content.additional_properties + run_has_reasoning_text = run_has_reasoning_text or _has_reasoning_text_marker(content.additional_properties) flush() contents.clear() contents.extend(coalesced_contents) @@ -2524,41 +2793,19 @@ def _content_items_text(items: Any) -> str | None: return "".join(text_parts) -def _content_items_are_complete_value(items: Any) -> bool: - """Return True when items are marked as an authoritative full value. - - Providers that resend a complete snapshot after incremental deltas should set - ``additional_properties["stream_complete"]`` (or ``"complete"``) on the text - content. Core only looks at that generic flag -- not provider event shapes -- - so ordinary deltas always concatenate even when one text is a string prefix of - the other (see #8955 / #8903). - """ - if not isinstance(items, list): - return False - for item in cast(list[object], items): - if not isinstance(item, Content): - continue - props = item.additional_properties or {} - if props.get("stream_complete") or props.get("complete"): - return True - return False - - -def _merge_content_item_lists(existing: Any, incoming: Any) -> Any: +def _merge_content_item_lists(existing: Any, incoming: Any, *, replace: bool) -> Any: """Merge streamed nested content lists. - Incremental text deltas are always concatenated. A later complete/full value - (``stream_complete`` / ``complete`` on ``additional_properties``) replaces the - accumulation. + Incremental values append. A content update with ``merge_mode="replace"`` + supplies an authoritative snapshot. """ + if replace: + return deepcopy(incoming) if incoming is None: return existing if existing is None: return deepcopy(incoming) - if _content_items_are_complete_value(incoming): - return deepcopy(incoming) - # Empty accumulation (e.g. initial outputs=[]) keeps the first real payload. if isinstance(existing, list) and len(cast(list[object], existing)) == 0: return deepcopy(incoming) @@ -2566,10 +2813,8 @@ def _merge_content_item_lists(existing: Any, incoming: Any) -> Any: existing_text = _content_items_text(existing) incoming_text = _content_items_text(incoming) if existing_text is not None and incoming_text is not None: - existing_items = cast(list[Content], existing) - merged = deepcopy(existing_items[0]) - merged.text = existing_text + incoming_text - return [merged] + text_items = [*cast(list[Content], existing), *cast(list[Content], incoming)] + return [_merge_content_run(text_items, "text")] if isinstance(existing, list) and isinstance(incoming, list): existing_list = cast(list[object], existing) @@ -2580,10 +2825,20 @@ def _merge_content_item_lists(existing: Any, incoming: Any) -> Any: def _merge_code_interpreter_content(existing: Content, incoming: Content) -> None: """Merge two code interpreter content items for the same logical call.""" - existing.inputs = _merge_content_item_lists(existing.inputs, incoming.inputs) - existing.outputs = _merge_content_item_lists(existing.outputs, incoming.outputs) + if existing.call_id and incoming.call_id and existing.call_id != incoming.call_id: + raise AdditionItemMismatch("Cannot merge code interpreter content with different call_ids") + replace = incoming.merge_mode == "replace" + existing.call_id = existing.call_id or incoming.call_id + existing.inputs = _merge_content_item_lists(existing.inputs, incoming.inputs, replace=replace) + existing.outputs = _merge_content_item_lists(existing.outputs, incoming.outputs, replace=replace) + existing.status = incoming.status if incoming.status is not None else existing.status + existing.merge_mode = incoming.merge_mode existing.annotations = _combine_annotations(existing.annotations, incoming.annotations) - existing.additional_properties = {**existing.additional_properties, **incoming.additional_properties} + existing.additional_properties = merge_additional_properties( + existing.additional_properties, + incoming.additional_properties, + incoming_wins=True, + ) existing.raw_representation = _combine_raw_representations(existing.raw_representation, incoming.raw_representation) @@ -2591,7 +2846,12 @@ def _code_interpreter_key(content: Content) -> tuple[str, str] | None: """Return the aggregation key for code interpreter call/result content.""" if content.type not in {"code_interpreter_tool_call", "code_interpreter_tool_result"}: return None - call_id = content.call_id or content.additional_properties.get("item_id") + call_id = content.call_id or get_namespaced_value( + content.additional_properties, + "openai", + "item_id", + content.additional_properties.get("item_id"), + ) if not isinstance(call_id, str) or not call_id: return None return content.type, call_id @@ -2630,6 +2890,23 @@ def _finalize_response(response: ChatResponse | AgentResponse) -> None: _coalesce_text_content(msg.contents, "text_reasoning") _coalesce_code_interpreter_content(msg.contents) _coalesce_function_call_occurrences(response) + for message in response.messages: + for content in message.contents: + _clear_merge_mode(content) + + +def _clear_merge_mode(content: Content) -> None: + """Clear update-only merge state from finalized response content.""" + content.merge_mode = None + for value in (content.items, content.inputs, content.outputs): + if isinstance(value, Sequence) and not isinstance(value, (str, bytes)): + for item in cast("Sequence[Any]", value): + if isinstance(item, Content): + _clear_merge_mode(item) + if isinstance(content.function_call, Content): + _clear_merge_mode(content.function_call) + if isinstance(content.screenshot, Content): + _clear_merge_mode(content.screenshot) def _coalesce_function_call_occurrences(response: ChatResponse | AgentResponse) -> None: @@ -2716,16 +2993,14 @@ def _last_non_empty_assistant_message_text(messages: Sequence[Message]) -> str: if message.role != "assistant": continue if any( - content.type == "text" - and content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) == _MODEL_OUTPUT_REFUSAL + content.type == "text" and _model_output_kind(content.additional_properties) == _MODEL_OUTPUT_REFUSAL for content in message.contents ): return "" text = "".join( (content.text or "") for content in message.contents - if content.type == "text" - and content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) != _MODEL_OUTPUT_REFUSAL + if content.type == "text" and _model_output_kind(content.additional_properties) != _MODEL_OUTPUT_REFUSAL ) if text.strip(): return text @@ -3082,6 +3357,7 @@ class ChatResponseUpdate(SerializationMixin): to the resulting `Message` objects. response_id: The ID of the response of which this update is a part. message_id: The ID of the message of which this update is a part. + merge_mode: Whether this update appends to or replaces the current logical message contents. conversation_id: An identifier for the state of the conversation of which this update is a part. model: The model associated with this response update. created_at: A timestamp for the chat response update. @@ -3127,6 +3403,7 @@ def __init__( author_name: str | None = None, response_id: str | None = None, message_id: str | None = None, + merge_mode: Literal["append", "replace"] | None = None, conversation_id: str | None = None, model: str | None = None, created_at: CreatedAtT | None = None, @@ -3143,6 +3420,7 @@ def __init__( author_name: Optional name of the author of the response update. response_id: Optional ID of the response of which this update is a part. message_id: Optional ID of the message of which this update is a part. + merge_mode: Whether this update appends to or replaces the current logical message contents. conversation_id: Optional identifier for the state of the conversation of which this update is a part model: Optional model associated with this response update. created_at: Optional timestamp for the chat response update. @@ -3154,6 +3432,9 @@ def __init__( from an underlying implementation. """ + if merge_mode not in (None, "append", "replace"): + raise ValueError(f"Invalid merge mode: {merge_mode!r}") + # Handle contents - support dict conversion for from_dict if contents is None: self.contents: list[Content] = [] @@ -3172,6 +3453,7 @@ def __init__( self.author_name = author_name self.response_id = response_id self.message_id = message_id + self.merge_mode: Literal["append", "replace"] | None = merge_mode self.conversation_id = conversation_id self.model = model self.created_at = created_at @@ -3561,6 +3843,7 @@ class AgentResponseUpdate(SerializationMixin): scenarios to track which agent generated specific parts of the response. response_id: The ID of the response of which this update is a part. message_id: The ID of the message of which this update is a part. + merge_mode: Whether this update appends to or replaces the current logical message contents. created_at: A timestamp for the response update. additional_properties: Any additional properties associated with the update. raw_representation: The raw representation from an underlying implementation. @@ -3606,6 +3889,7 @@ def __init__( agent_id: str | None = None, response_id: str | None = None, message_id: str | None = None, + merge_mode: Literal["append", "replace"] | None = None, created_at: CreatedAtT | None = None, finish_reason: FinishReasonLiteral | FinishReason | None = None, continuation_token: ContinuationToken | None = None, @@ -3622,6 +3906,7 @@ def __init__( agent_id: Optional identifier of the agent that produced this update. response_id: Optional ID of the response of which this update is a part. message_id: Optional ID of the message of which this update is a part. + merge_mode: Whether this update appends to or replaces the current logical message contents. created_at: Optional timestamp for the chat response update. finish_reason: The reason the model stopped generating. Common values include ``"stop"`` (natural completion), ``"length"`` (token limit), and @@ -3632,6 +3917,9 @@ def __init__( raw_representation: Optional raw representation of the chat response update. """ + if merge_mode not in (None, "append", "replace"): + raise ValueError(f"Invalid merge mode: {merge_mode!r}") + # Handle contents - support dict conversion for from_dict if contents is None: self.contents: list[Content] = [] @@ -3651,6 +3939,7 @@ def __init__( self.agent_id = agent_id self.response_id = response_id self.message_id = message_id + self.merge_mode: Literal["append", "replace"] | None = merge_mode self.created_at = created_at self.finish_reason = finish_reason self.continuation_token = continuation_token @@ -3707,6 +3996,7 @@ def map_chat_to_agent_update(update: ChatResponseUpdate, agent_name: str | None) author_name=update.author_name or agent_name, response_id=update.response_id, message_id=update.message_id, + merge_mode=getattr(update, "merge_mode", None), created_at=update.created_at, finish_reason=update.finish_reason, # type: ignore[arg-type] continuation_token=update.continuation_token, diff --git a/python/packages/core/agent_framework/_workflows/_agent.py b/python/packages/core/agent_framework/_workflows/_agent.py index 197b7298aec..cee5d96abd7 100644 --- a/python/packages/core/agent_framework/_workflows/_agent.py +++ b/python/packages/core/agent_framework/_workflows/_agent.py @@ -31,6 +31,7 @@ UsageDetails, _copy_operation_state, # pyright: ignore[reportPrivateUsage] add_usage_details, + merge_additional_properties, ) from ..exceptions import AgentException, AgentInvalidRequestException, AgentInvalidResponseException from ._checkpoint import CheckpointStorage @@ -956,7 +957,10 @@ def _add_raw(value: object) -> None: created_at=incoming.created_at or current.created_at, usage_details=add_usage_details(current.usage_details, incoming.usage_details), raw_representation=raw_list if raw_list else None, - additional_properties=incoming.additional_properties or current.additional_properties, + additional_properties=merge_additional_properties( + current.additional_properties, + incoming.additional_properties, + ), ) # PHASE 2: CONVERT GROUPED UPDATES TO RESPONSES AND MERGE @@ -995,9 +999,10 @@ def _add_raw(value: object) -> None: ): latest_created_at = aggregated.created_at if aggregated.additional_properties: - if merged_additional_properties is None: - merged_additional_properties = {} - merged_additional_properties.update(aggregated.additional_properties) + merged_additional_properties = merge_additional_properties( + merged_additional_properties, + aggregated.additional_properties, + ) raw_value = aggregated.raw_representation if raw_value: cast_value = cast(object | list[object], raw_value) @@ -1019,9 +1024,10 @@ def _add_raw(value: object) -> None: ): latest_created_at = flattened.created_at if flattened.additional_properties: - if merged_additional_properties is None: - merged_additional_properties = {} - merged_additional_properties.update(flattened.additional_properties) + merged_additional_properties = merge_additional_properties( + merged_additional_properties, + flattened.additional_properties, + ) flat_raw = flattened.raw_representation if flat_raw: cast_flat = cast(object | list[object], flat_raw) diff --git a/python/packages/core/agent_framework/_workflows/_agent_executor.py b/python/packages/core/agent_framework/_workflows/_agent_executor.py index da5c3f66c8b..eb81f81572e 100644 --- a/python/packages/core/agent_framework/_workflows/_agent_executor.py +++ b/python/packages/core/agent_framework/_workflows/_agent_executor.py @@ -641,10 +641,10 @@ async def _cancel_pending_request( ): cancellation_result = Content.from_function_result( call_id=cancelled_request.call_id, + id=cancelled_request.id, result="Error: Tool call was cancelled.", additional_properties={"cancelled": True}, ) - cancellation_result.id = cancelled_request.id self._queue_pending_response(request_id, cancellation_result) elif cancelled_request is not None and cancelled_request.type == "computer_tool_call": self._queue_pending_response( diff --git a/python/packages/core/tests/core/test_function_invocation_logic.py b/python/packages/core/tests/core/test_function_invocation_logic.py index bd48765f275..f37f6a8848f 100644 --- a/python/packages/core/tests/core/test_function_invocation_logic.py +++ b/python/packages/core/tests/core/test_function_invocation_logic.py @@ -2095,6 +2095,34 @@ def func_no_approval(arg1: str) -> str: assert response.messages[0].contents == [informational_call] +async def test_hosted_function_call_is_not_invoked(chat_client_base: SupportsChatGetResponse): + exec_counter = 0 + + @tool(name="provider_tool", approval_mode="never_require") + def provider_tool(arg1: str) -> str: + nonlocal exec_counter + exec_counter += 1 + return f"Processed {arg1}" + + hosted_call = Content.from_function_call( + call_id="1", + name="provider_tool", + arguments='{"arg1": "value1"}', + hosted=True, + ) + chat_client_base.run_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + ChatResponse(messages=Message(role="assistant", contents=[hosted_call])) + ] + + response = await chat_client_base.get_response( + [Message(role="user", contents=["hello"])], + options={"tool_choice": "auto", "tools": [provider_tool]}, + ) + + assert exec_counter == 0 + assert response.messages[0].contents == [hosted_call] + + async def test_only_actionable_function_calls_are_invoked(chat_client_base: SupportsChatGetResponse): exec_counter = 0 @@ -10364,6 +10392,53 @@ async def func2(arg1: str) -> str: assert len(results) == 2 +async def test_streaming_function_loop_sums_usage_across_provider_turns( + chat_client_base: SupportsChatGetResponse, +) -> None: + @tool(name="get_value", approval_mode="never_require") + def get_value() -> str: + return "value" + + chat_client_base.streaming_responses = [ # type: ignore[attr-defined] # ty: ignore[unresolved-attribute] + [ + ChatResponseUpdate( + contents=[Content.from_function_call(call_id="call_1", name="get_value", arguments="{}")], + role="assistant", + ), + ChatResponseUpdate( + contents=[ + Content.from_usage( + {"input_token_count": 3, "output_token_count": 2}, + merge_mode="replace", + ) + ], + role="assistant", + ), + ], + [ + ChatResponseUpdate(contents=[Content.from_text("done")], role="assistant"), + ChatResponseUpdate( + contents=[ + Content.from_usage( + {"input_token_count": 5, "output_token_count": 4}, + merge_mode="replace", + ) + ], + role="assistant", + ), + ], + ] + + stream = chat_client_base.get_response( + [Message(role="user", contents=["hello"])], + options={"tool_choice": "auto", "tools": [get_value]}, + stream=True, + ) + response = await stream.get_final_response() + + assert response.usage_details == {"input_token_count": 8, "output_token_count": 6} + + async def test_streaming_approval_requests_in_assistant_message(chat_client_base: SupportsChatGetResponse): """Approval requests should be added to assistant updates in streaming mode.""" exec_counter = 0 @@ -12895,8 +12970,10 @@ async def test_hosted_approval_response_passes_through_without_session( name="hosted_tool", arguments={}, id="hosted-occurrence", + hosted=True, + server_name="hosted_server", + additional_properties={"server_label": "hosted_server"}, ) - hosted_call.additional_properties["server_label"] = "hosted_server" hosted_response = Content.from_function_approval_response( id="hosted-occurrence", function_call=hosted_call, diff --git a/python/packages/core/tests/core/test_harness_tool_approval.py b/python/packages/core/tests/core/test_harness_tool_approval.py index 85416e6fa19..24db9cad0b5 100644 --- a/python/packages/core/tests/core/test_harness_tool_approval.py +++ b/python/packages/core/tests/core/test_harness_tool_approval.py @@ -2947,13 +2947,17 @@ def hosted_tool() -> str: calls += 1 return "hosted" - def hosted_call(call_id: str, server_label: str) -> Content: - return Content.from_function_call( + def hosted_approval_request(call_id: str, server_label: str) -> Content: + function_call = Content.from_function_call( call_id=call_id, name="hosted_tool", arguments="{}", + id=call_id, + hosted=True, + server_name=server_label, additional_properties={"server_label": server_label}, ) + return Content.from_function_approval_request(id=call_id, function_call=function_call) agent = Agent( client=chat_client_base, @@ -2962,7 +2966,7 @@ def hosted_call(call_id: str, server_label: str) -> Content: ) session = AgentSession(session_id="hosted-boundary-session") chat_client_base.run_responses = [ - ChatResponse(messages=Message(role="assistant", contents=[hosted_call("call_initial", "server-a")])) + ChatResponse(messages=Message(role="assistant", contents=[hosted_approval_request("call_initial", "server-a")])) ] first_response = await agent.run("call hosted a", session=session) @@ -2974,7 +2978,12 @@ def hosted_call(call_id: str, server_label: str) -> Content: assert calls == 0 chat_client_base.run_responses = [ - ChatResponse(messages=Message(role="assistant", contents=[hosted_call("call_same_server", "server-a")])), + ChatResponse( + messages=Message( + role="assistant", + contents=[hosted_approval_request("call_same_server", "server-a")], + ) + ), ChatResponse(messages=Message(role="assistant", contents=["same server done"])), ] @@ -2985,7 +2994,12 @@ def hosted_call(call_id: str, server_label: str) -> Content: assert calls == 0 chat_client_base.run_responses = [ - ChatResponse(messages=Message(role="assistant", contents=[hosted_call("call_other_server", "server-b")])) + ChatResponse( + messages=Message( + role="assistant", + contents=[hosted_approval_request("call_other_server", "server-b")], + ) + ) ] other_server_response = await agent.run("call hosted b", session=session) diff --git a/python/packages/core/tests/core/test_sessions.py b/python/packages/core/tests/core/test_sessions.py index e23d560e284..9e01176cc3f 100644 --- a/python/packages/core/tests/core/test_sessions.py +++ b/python/packages/core/tests/core/test_sessions.py @@ -673,13 +673,9 @@ async def test_after_run_stores_coalesced_code_interpreter_chunks(self) -> None: contents=[ Content.from_code_interpreter_tool_call( call_id="ci_123", - # Final snapshot: authoritative full value via stream_complete. - inputs=[ - Content.from_text( - text="import pandas as pd", - additional_properties={"stream_complete": True}, - ) - ], + # Final snapshot: authoritative full value. + inputs=[Content.from_text(text="import pandas as pd")], + merge_mode="replace", additional_properties={"sequence_number": 3}, ) ], diff --git a/python/packages/core/tests/core/test_types.py b/python/packages/core/tests/core/test_types.py index 2c41acd93b4..bba799f339a 100644 --- a/python/packages/core/tests/core/test_types.py +++ b/python/packages/core/tests/core/test_types.py @@ -4,6 +4,7 @@ import base64 import contextlib import json +import pickle import warnings from collections.abc import AsyncIterable, AsyncIterator, Awaitable, Callable, Sequence from copy import copy, deepcopy @@ -32,8 +33,11 @@ ToolMode, UsageDetails, detect_media_type_from_base64, + get_namespaced_value, + merge_additional_properties, merge_chat_options, tool, + with_namespaced_value, ) from agent_framework._compaction import ( GROUP_ANNOTATION_KEY, @@ -168,6 +172,26 @@ def test_marked_refusal_text_is_excluded_from_structured_output() -> None: assert response.value is None +def test_namespaced_refusal_text_is_excluded_from_structured_output() -> None: + response = ChatResponse( + messages=[ + Message( + "assistant", + [ + Content.from_text( + '{"should_not": "parse"}', + additional_properties={"core": {"model_output_kind": "refusal"}}, + ) + ], + ) + ], + response_format={"type": "object"}, + ) + + assert response.text == '{"should_not": "parse"}' + assert response.value is None + + @pytest.mark.parametrize("response_type", [ChatResponse, AgentResponse]) def test_final_marked_refusal_does_not_fall_back_to_earlier_structured_output(response_type: type) -> None: response = response_type( @@ -454,7 +478,7 @@ def test_code_interpreter_tool_call_stream_concatenates_indent_prefix_deltas() - def test_code_interpreter_tool_call_stream_done_replaces_accumulated_deltas() -> None: - """A stream_complete snapshot is authoritative and replaces prior delta text.""" + """An explicit replace snapshot is authoritative over prior deltas.""" updates = [ ChatResponseUpdate( contents=[ @@ -465,12 +489,8 @@ def test_code_interpreter_tool_call_stream_done_replaces_accumulated_deltas() -> contents=[ Content.from_code_interpreter_tool_call( call_id="ci_1", - inputs=[ - Content.from_text( - text="import os\nimport sys", - additional_properties={"stream_complete": True}, - ) - ], + inputs=[Content.from_text(text="import os\nimport sys")], + merge_mode="replace", ) ] ), @@ -514,6 +534,46 @@ def test_code_interpreter_tool_result_content_outputs(): assert result.outputs[1].type == "uri" +def test_code_interpreter_tool_result_replace_snapshot() -> None: + first = Content.from_code_interpreter_tool_result( + call_id="ci_1", + outputs=[Content.from_text("delta")], + status="in_progress", + additional_properties={"openai": {"left": 1, "item_id": "old"}}, + ) + replacement = Content.from_code_interpreter_tool_result( + call_id="ci_1", + outputs=[Content.from_text("snapshot", merge_mode="replace")], + status="completed", + merge_mode="replace", + additional_properties={"openai": {"right": 2, "item_id": "new"}}, + ) + + result = first + replacement + assert [item.text for item in result.outputs or []] == ["snapshot"] + assert result.status == "completed" + assert result.additional_properties["openai"] == {"left": 1, "right": 2, "item_id": "new"} + + +def test_finalized_response_clears_merge_modes_recursively() -> None: + response = ChatResponse.from_updates([ + ChatResponseUpdate( + contents=[ + Content.from_code_interpreter_tool_call( + call_id="ci_1", + inputs=[Content.from_text("print('ok')", merge_mode="replace")], + merge_mode="replace", + ) + ] + ) + ]) + + content = response.messages[0].contents[0] + assert content.merge_mode is None + assert content.inputs is not None + assert content.inputs[0].merge_mode is None + + # region: Image generation content @@ -791,6 +851,89 @@ def test_function_call_content(): assert isinstance(content, Content) +def test_content_merge_mode_validation_and_serialization() -> None: + content = Content.from_text("snapshot", merge_mode="replace") + + assert content.to_dict()["merge_mode"] == "replace" + assert Content.from_dict(content.to_dict()).merge_mode == "replace" + + with pytest.raises(ValueError, match="Invalid merge mode"): + Content("text", merge_mode=cast(Any, "invalid")) + with pytest.raises(ValueError, match="does not support merge_mode"): + Content("function_result", call_id="call", result="done", merge_mode="append") + + +def test_function_call_hosted_serialization_and_legacy_inference() -> None: + local = Content.from_function_call(call_id="local", name="f") + assert local.hosted is False + assert local.to_dict()["hosted"] is False + + described_local = Content.from_function_call(call_id="described-local", name="f", server_name="provider") + restored_described_local = Content.from_dict(described_local.to_dict()) + assert restored_described_local.hosted is False + assert restored_described_local.server_name == "provider" + + hosted = Content.from_function_call( + call_id="hosted", + name="f", + hosted=True, + server_name="provider", + additional_properties={"server_label": "provider"}, + ) + restored = Content.from_dict(hosted.to_dict()) + assert restored.hosted is True + assert restored.server_name == "provider" + + with pytest.warns(FutureWarning, match="legacy.*server_label"): + legacy = Content.from_dict({ + "type": "function_call", + "call_id": "legacy", + "name": "f", + "additional_properties": {"server_label": "provider"}, + }) + assert legacy.hosted is True + assert legacy.server_name == "provider" + + with pytest.raises(ValueError, match="hosted must be true or false"): + Content.from_dict({ + "type": "function_call", + "call_id": "invalid", + "name": "f", + "hosted": "yes", + }) + with pytest.raises(ValueError, match="hosted must be true or false"): + Content.from_dict({ + "type": "function_call", + "call_id": "invalid-null", + "name": "f", + "hosted": None, + }) + with pytest.raises(ValueError, match="conflicts"): + Content.from_function_call( + call_id="conflict", + name="f", + hosted=True, + server_name="one", + additional_properties={"server_label": "two"}, + ) + + +def test_function_call_legacy_hosted_inference_when_unpickling() -> None: + content = Content.from_function_call(call_id="legacy", name="f") + state = content.__getstate__() + state.pop("hosted") + state["server_name"] = None + state["additional_properties"] = {"server_label": "provider"} + + restored = Content.__new__(Content) + with pytest.warns(FutureWarning, match="legacy.*server_label"): + restored.__setstate__(state) + + assert restored.hosted is True + assert restored.server_name == "provider" + assert pickle.loads(pickle.dumps(restored)).hosted is True + + def test_function_call_content_informational_only_serialization(): content = Content.from_function_call( call_id="1", @@ -877,6 +1020,51 @@ def test_function_call_content_add_merging_and_errors(): b = Content("function_call", call_id="1", name=None, arguments="") assert (a + b).name is None + # Hosted routing is fail-safe across fragments, and the descriptive boundary is retained. + a = Content.from_function_call(call_id="1", name="f", arguments="", hosted=False) + b = Content.from_function_call( + call_id="1", + name="f", + arguments="{}", + hosted=True, + server_name="provider", + ) + c = a + b + assert c.hosted is True + assert c.server_name == "provider" + + # A complete snapshot replaces streamed arguments while retaining stable identity. + a = Content.from_function_call(call_id="1", name="f", arguments='{"x":', id="occurrence") + b = Content.from_function_call( + call_id="1", + name="f", + arguments='{"x": 1}', + id="occurrence", + merge_mode="replace", + ) + c = a + b + assert c.arguments == '{"x": 1}' + assert c.id == "occurrence" + + # A complete snapshot replaces the semantic name while retaining fail-safe hosted routing. + a = Content.from_function_call( + call_id="1", + name="old_tool", + arguments="{}", + id="occurrence", + hosted=True, + ) + b = Content.from_function_call( + call_id="1", + name="new_tool", + arguments="{}", + id="occurrence", + merge_mode="replace", + ) + c = a + b + assert c.name == "new_tool" + assert c.hosted is True + # region FunctionResultContent @@ -898,6 +1086,20 @@ def test_function_result_content(): assert isinstance(content, Content) +def test_function_result_occurrence_identity_and_status_roundtrip() -> None: + content = Content.from_function_result( + call_id="provider-call", + id="occurrence", + result="done", + status="completed", + ) + + restored = Content.from_dict(content.to_dict()) + assert restored.call_id == "provider-call" + assert restored.id == "occurrence" + assert restored.status == "completed" + + # region UsageDetails @@ -1072,6 +1274,8 @@ def test_hosted_function_approval_request_allows_provider_request_identity_witho call_id="provider-call", name="hosted", id="af-call-occurrence", + hosted=True, + server_name="provider", additional_properties={"server_label": "provider"}, ) @@ -1511,6 +1715,45 @@ def test_chat_response_updates_to_chat_response_two(): assert chat_response.messages[1].message_id == "2" +@mark.parametrize( + ("response_type", "update_type"), + [(ChatResponse, ChatResponseUpdate), (AgentResponse, AgentResponseUpdate)], +) +def test_response_update_replace_replaces_logical_message_contents(response_type: Any, update_type: Any) -> None: + updates = [ + update_type( + contents=[ + Content.from_text("old"), + Content.from_uri("https://example.com/old"), + ], + role="assistant", + message_id="message-1", + ), + update_type( + contents=[ + Content.from_uri("https://example.com/new"), + Content.from_data(b"new", media_type="application/octet-stream"), + ], + role="assistant", + message_id="message-1", + merge_mode="replace", + ), + ] + + response = response_type.from_updates(updates) + + assert len(response.messages) == 1 + assert [content.type for content in response.messages[0].contents] == ["uri", "data"] + assert updates[1].to_dict()["merge_mode"] == "replace" + assert update_type.from_dict(updates[1].to_dict()).merge_mode == "replace" + + +@mark.parametrize("update_type", [ChatResponseUpdate, AgentResponseUpdate]) +def test_response_update_rejects_invalid_merge_mode(update_type: Any) -> None: + with raises(ValueError, match="Invalid merge mode"): + update_type(merge_mode=cast(Any, "invalid")) + + def test_chat_response_updates_to_chat_response_multiple(): """Test converting ChatResponseUpdate to ChatResponse.""" # Create a Message @@ -2457,6 +2700,59 @@ def test_text_content_add_comprehensive_coverage(): assert result.raw_representation == ["raw1", "raw2", "raw3"] +def test_namespaced_additional_properties_helpers_copy_and_merge_one_level() -> None: + source: dict[str, Any] = { + "openai": {"item_id": "item-1"}, + "application": {"left": 1}, + } + content = Content.from_text("hello", additional_properties=source) + source["openai"]["item_id"] = "mutated" + + assert get_namespaced_value(content.additional_properties, "openai", "item_id") == "item-1" + assert get_namespaced_value(content.additional_properties, "missing", "key", "fallback") == "fallback" + + updated = with_namespaced_value(content.additional_properties, "openai", "status", "completed") + assert updated["openai"] == {"item_id": "item-1", "status": "completed"} + assert content.additional_properties["openai"] == {"item_id": "item-1"} + + existing_wins = merge_additional_properties( + {"openai": {"item_id": "old", "left": 1}, "flat": "old"}, + {"openai": {"item_id": "new", "right": 2}, "flat": "new"}, + incoming_wins=False, + ) + assert existing_wins == { + "openai": {"item_id": "old", "left": 1, "right": 2}, + "flat": "old", + } + + incoming_wins = merge_additional_properties( + {"openai": {"item_id": "old", "left": 1}, "flat": "old"}, + {"openai": {"item_id": "new", "right": 2}, "flat": "new"}, + ) + assert incoming_wins == { + "openai": {"item_id": "new", "left": 1, "right": 2}, + "flat": "new", + } + + +def test_text_content_merge_modes_and_metadata_winner() -> None: + first = Content.from_text( + "delta", + merge_mode="append", + additional_properties={"openai": {"item_id": "old", "left": 1}}, + ) + replacement = Content.from_text( + "snapshot", + merge_mode="replace", + additional_properties={"openai": {"item_id": "new", "right": 2}}, + ) + + result = first + replacement + assert result.text == "snapshot" + assert result.merge_mode == "replace" + assert result.additional_properties["openai"] == {"item_id": "old", "left": 1, "right": 2} + + def test_text_content_iadd_coverage(): """Test TextContent += operator for better coverage.""" @@ -2540,6 +2836,21 @@ def test_text_reasoning_content_add_coverage(): assert result.text == "Thinking 1 Thinking 2" +def test_text_reasoning_replace_resets_semantic_payload() -> None: + first = Content.from_text_reasoning(text="delta", protected_data="old", status="in_progress") + replacement = Content.from_text_reasoning( + text="snapshot", + protected_data="new", + status="completed", + merge_mode="replace", + ) + + result = first + replacement + assert result.text == "snapshot" + assert result.protected_data == "new" + assert result.status == "completed" + + def test_text_reasoning_content_iadd_coverage(): """Test TextReasoningContent += operator for better coverage.""" @@ -2891,6 +3202,26 @@ def test_agent_response_from_updates_preserves_refusal_marker() -> None: assert response.text == "I cannot help." +def test_canonical_reasoning_marker_preserves_summary_boundary() -> None: + response = AgentResponse.from_updates([ + AgentResponseUpdate( + contents=[ + Content.from_text_reasoning( + text="summary", + additional_properties={"openai": {"reasoning_text": True}}, + ) + ], + role="assistant", + ), + AgentResponseUpdate( + contents=[Content.from_text_reasoning(text="private reasoning")], + role="assistant", + ), + ]) + + assert [content.text for content in response.messages[0].contents] == ["summary", "private reasoning"] + + @pytest.mark.parametrize( ("updates", "expected"), [ @@ -4357,6 +4688,46 @@ def test_content_add_usage_content(): assert "raw2" in result.raw_representation +def test_content_replace_usage_content(): + usage1 = Content.from_usage( + {"input_token_count": 100, "output_token_count": 50}, + merge_mode="append", + ) + usage2 = Content.from_usage( + {"input_token_count": 20, "output_token_count": 10}, + merge_mode="replace", + ) + + result = usage1 + usage2 + + assert result.usage_details == {"input_token_count": 20, "output_token_count": 10} + assert result.merge_mode == "replace" + + +def test_response_usage_replace_snapshot() -> None: + response = ChatResponse.from_updates([ + ChatResponseUpdate(contents=[Content.from_usage({"input_token_count": 100}, merge_mode="append")]), + ChatResponseUpdate(contents=[Content.from_usage({"input_token_count": 20}, merge_mode="replace")]), + ]) + + assert response.usage_details == {"input_token_count": 20} + + +def test_response_metadata_merges_namespaces_with_incoming_values_winning() -> None: + response = ChatResponse.from_updates([ + ChatResponseUpdate( + contents=[Content.from_text("one")], + additional_properties={"openai": {"item_id": "old", "left": 1}}, + ), + ChatResponseUpdate( + contents=[Content.from_text("two")], + additional_properties={"openai": {"item_id": "new", "right": 2}}, + ), + ]) + + assert response.additional_properties["openai"] == {"item_id": "new", "left": 1, "right": 2} + + def test_content_add_usage_content_with_none_raw_representation(): """Test adding usage content when one has None raw_representation.""" usage1 = Content( @@ -6670,11 +7041,13 @@ def test_map_chat_to_agent_update_forwards_finish_reason() -> None: chat_update = ChatResponseUpdate( contents=[Content.from_text("test")], finish_reason="length", + merge_mode="replace", ) agent_update = map_chat_to_agent_update(chat_update, agent_name="test_agent") assert agent_update.finish_reason == "length" assert agent_update.author_name == "test_agent" + assert agent_update.merge_mode == "replace" def test_process_update_propagates_finish_reason_to_agent_response() -> None: diff --git a/python/packages/core/tests/workflow/test_workflow_agent.py b/python/packages/core/tests/workflow/test_workflow_agent.py index 92f7bcfa545..97ee0433694 100644 --- a/python/packages/core/tests/workflow/test_workflow_agent.py +++ b/python/packages/core/tests/workflow/test_workflow_agent.py @@ -1932,8 +1932,7 @@ def test_merge_updates_metadata_aggregation(self): "source": "executor2", # From resp-2 (latest source value) "priority": "high", # From resp-1 first update (resp-1 processed before resp-2) "category": "analysis", # From resp-2 (only place this appears) - # "details": "merged" is NOT in final result because resp-1's aggregated - # properties only include final merged result from its own updates + "details": "merged", # Preserved from the later resp-1 update } assert result.additional_properties == expected_properties diff --git a/python/packages/devui/AGENTS.md b/python/packages/devui/AGENTS.md index 18c121ceb46..566dcfee0ae 100644 --- a/python/packages/devui/AGENTS.md +++ b/python/packages/devui/AGENTS.md @@ -15,9 +15,10 @@ Interactive developer UI for testing and debugging agents and workflows. - **`OpenAIResponse`** / **`OpenAIError`** - OpenAI-compatible response models - **`DiscoveryResponse`** / **`EntityInfo`** - Entity discovery models -Text content carrying `additional_properties["model_output_kind"] == "refusal"` is mapped to native -Responses refusal parts and events. Mapper aggregation, live rendering, and recovery state retain -text/refusal boundaries by item and content index. +Text content carrying `additional_properties["core"]["model_output_kind"] == "refusal"` is mapped to native +Responses refusal parts and events; the legacy flat key remains readable. OpenAI input metadata uses the `openai` +namespace while replay-critical writers retain documented legacy keys. Mapper aggregation, live rendering, and +recovery state retain text/refusal boundaries by item and content index. ## Usage diff --git a/python/packages/devui/agent_framework_devui/_conversations.py b/python/packages/devui/agent_framework_devui/_conversations.py index 76a8f348eb1..1c8b908b104 100644 --- a/python/packages/devui/agent_framework_devui/_conversations.py +++ b/python/packages/devui/agent_framework_devui/_conversations.py @@ -14,7 +14,7 @@ from collections.abc import MutableSequence from typing import Any, Literal, cast -from agent_framework import AgentSession, Content, Message +from agent_framework import AgentSession, Content, Message, get_namespaced_value from agent_framework._workflows._checkpoint import InMemoryCheckpointStorage, WorkflowCheckpoint from openai.types.conversations import Conversation, ConversationDeletedResource from openai.types.conversations.conversation_item import ConversationItem @@ -34,6 +34,14 @@ _MODEL_OUTPUT_KIND_KEY = "model_output_kind" _MODEL_OUTPUT_REFUSAL = "refusal" + +def _refusal_additional_properties() -> dict[str, Any]: + return { + "core": {_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + _MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL, + } + + # Type alias for OpenAI Message role literals MessageRole = Literal["unknown", "user", "assistant", "system", "critic", "discriminator", "developer", "tool"] @@ -541,12 +549,16 @@ def _to_agent_content(content: dict[str, Any]) -> Content | None: return None return Content.from_text( text=refusal, - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + additional_properties=_refusal_additional_properties(), ) if content_type == "input_image": detail = content.get("detail", "auto") image_properties = { + "openai": { + "content_type": "input_image", + "image_detail": detail if isinstance(detail, str) else "auto", + }, "openai_content_type": "input_image", "detail": detail if isinstance(detail, str) else "auto", } @@ -569,7 +581,10 @@ def _to_agent_content(content: dict[str, Any]) -> Content | None: filename = content.get("filename") filename = filename if isinstance(filename, str) else "" detail = content.get("detail") - file_properties: dict[str, Any] = {"openai_content_type": "input_file"} + file_properties: dict[str, Any] = { + "openai": {"content_type": "input_file"}, + "openai_content_type": "input_file", + } if filename: file_properties["filename"] = filename if isinstance(detail, str): @@ -586,6 +601,7 @@ def _to_agent_content(content: dict[str, Any]) -> Content | None: return Content.from_uri( uri=uri, media_type=media_type, + name=filename or None, additional_properties=file_properties, ) @@ -593,6 +609,7 @@ def _to_agent_content(content: dict[str, Any]) -> Content | None: if isinstance(file_url, str) and file_url: return Content.from_uri( uri=file_url, + name=filename or None, media_type=infer_media_type( filename=filename, uri=file_url, @@ -619,19 +636,37 @@ def _to_openai_content( """Convert one supported Agent Framework message part.""" content_type = content.type if content_type == "text": - if content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) == _MODEL_OUTPUT_REFUSAL: + if ( + get_namespaced_value( + content.additional_properties, + "core", + _MODEL_OUTPUT_KIND_KEY, + content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY), + ) + == _MODEL_OUTPUT_REFUSAL + ): return ResponseOutputRefusal(type="refusal", refusal=content.text or "") return TextContent(type="text", text=content.text or "") additional_properties = content.additional_properties or {} - openai_content_type = additional_properties.get("openai_content_type") + openai_content_type = get_namespaced_value( + additional_properties, + "openai", + "content_type", + additional_properties.get("openai_content_type"), + ) is_image = openai_content_type == "input_image" or ( openai_content_type is None and bool(content.media_type and content.media_type.startswith("image/")) ) if content_type in ("data", "uri"): if is_image: - detail = additional_properties.get("detail", "auto") + detail = get_namespaced_value( + additional_properties, + "openai", + "image_detail", + additional_properties.get("detail", "auto"), + ) if detail not in ("low", "high", "auto", "original"): detail = "auto" return ResponseInputImage( @@ -640,8 +675,13 @@ def _to_openai_content( detail=detail, ) - filename = additional_properties.get("filename") - detail = additional_properties.get("detail") + filename = content.name or additional_properties.get("filename") + detail = get_namespaced_value( + additional_properties, + "openai", + "image_detail", + additional_properties.get("detail"), + ) if detail not in ("low", "high", "auto"): detail = None return ResponseInputFile( @@ -653,7 +693,12 @@ def _to_openai_content( if content_type == "hosted_file": if is_image: - detail = additional_properties.get("detail", "auto") + detail = get_namespaced_value( + additional_properties, + "openai", + "image_detail", + additional_properties.get("detail", "auto"), + ) if detail not in ("low", "high", "auto", "original"): detail = "auto" return ResponseInputImage( @@ -663,7 +708,12 @@ def _to_openai_content( ) filename = content.name or additional_properties.get("filename") - detail = additional_properties.get("detail") + detail = get_namespaced_value( + additional_properties, + "openai", + "image_detail", + additional_properties.get("detail"), + ) if detail not in ("low", "high", "auto"): detail = None return ResponseInputFile( diff --git a/python/packages/devui/agent_framework_devui/_executor.py b/python/packages/devui/agent_framework_devui/_executor.py index 19c71146b00..f64a61964ec 100644 --- a/python/packages/devui/agent_framework_devui/_executor.py +++ b/python/packages/devui/agent_framework_devui/_executor.py @@ -721,6 +721,10 @@ def _convert_openai_input_to_chat_message( file_id = content_dict.get("file_id") detail = content_dict.get("detail", "auto") image_properties = { + "openai": { + "content_type": "input_image", + "image_detail": detail if isinstance(detail, str) else "auto", + }, "openai_content_type": "input_image", "detail": detail if isinstance(detail, str) else "auto", } @@ -754,10 +758,14 @@ def _convert_openai_input_to_chat_message( # Use file_data or file_url # Include filename in additional_properties for OpenAI/Azure file handling - file_properties: dict[str, Any] = {"openai_content_type": "input_file"} + file_properties: dict[str, Any] = { + "openai": {"content_type": "input_file"}, + "openai_content_type": "input_file", + } if filename: file_properties["filename"] = filename if isinstance(detail, str): + cast(dict[str, Any], file_properties["openai"])["image_detail"] = detail file_properties["detail"] = detail if isinstance(file_data, str) and file_data: @@ -775,6 +783,7 @@ def _convert_openai_input_to_chat_message( Content.from_uri( uri=data_uri, media_type=media_type, + name=filename or None, additional_properties=file_properties, ) ) @@ -788,6 +797,7 @@ def _convert_openai_input_to_chat_message( Content.from_uri( uri=file_url, media_type=media_type, + name=filename or None, additional_properties=file_properties, ) ) diff --git a/python/packages/devui/agent_framework_devui/_mapper.py b/python/packages/devui/agent_framework_devui/_mapper.py index 8476c1ac125..771a571e2aa 100644 --- a/python/packages/devui/agent_framework_devui/_mapper.py +++ b/python/packages/devui/agent_framework_devui/_mapper.py @@ -14,7 +14,7 @@ from typing import Any, Union, cast from uuid import uuid4 -from agent_framework import Content, Message, UsageDetails, add_usage_details +from agent_framework import Content, Message, UsageDetails, add_usage_details, get_namespaced_value from openai.types.responses import ( Response, ResponseContentPartAddedEvent, @@ -89,7 +89,16 @@ def _workflow_output_metadata(event_type: Any, executor_id: Any) -> dict[str, An def _is_refusal_text_content(content: Content) -> bool: - return content.type == "text" and content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) == _MODEL_OUTPUT_REFUSAL + return ( + content.type == "text" + and get_namespaced_value( + content.additional_properties, + "core", + _MODEL_OUTPUT_KIND_KEY, + content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY), + ) + == _MODEL_OUTPUT_REFUSAL + ) def _message_content_part(content: Content) -> ResponseOutputText | ResponseOutputRefusal: diff --git a/python/packages/devui/tests/devui/test_conversations.py b/python/packages/devui/tests/devui/test_conversations.py index 0983766c998..62f9e2dae6b 100644 --- a/python/packages/devui/tests/devui/test_conversations.py +++ b/python/packages/devui/tests/devui/test_conversations.py @@ -299,9 +299,20 @@ async def test_add_and_list_items_preserves_all_supported_message_parts(): stored_messages = store._conversations[conversation.id]["messages"] assert stored_messages[1].contents[0].media_type == "image/jpeg" + assert stored_messages[1].contents[0].additional_properties["openai"] == { + "content_type": "input_image", + "image_detail": "high", + } + assert stored_messages[1].contents[1].additional_properties["openai"] == { + "content_type": "input_image", + "image_detail": "low", + } assert stored_messages[1].contents[2].media_type == "application/pdf" + assert stored_messages[1].contents[2].additional_properties["openai"]["content_type"] == "input_file" assert stored_messages[1].contents[3].media_type == "audio/mpeg" + assert stored_messages[1].contents[3].additional_properties["openai"]["content_type"] == "input_file" assert stored_messages[1].contents[4].media_type == "image/jpeg" + assert stored_messages[1].contents[4].additional_properties["openai"]["content_type"] == "input_file" assert has_more is False listed_system, listed_user = listed_items diff --git a/python/packages/devui/tests/devui/test_mapper.py b/python/packages/devui/tests/devui/test_mapper.py index d050a80d2f6..d71a32c88ad 100644 --- a/python/packages/devui/tests/devui/test_mapper.py +++ b/python/packages/devui/tests/devui/test_mapper.py @@ -125,7 +125,7 @@ async def test_text_content_mapping(mapper: MessageMapper, test_request: AgentFr async def test_marked_refusal_text_mapping(mapper: MessageMapper, test_request: AgentFrameworkRequest) -> None: content = Content.from_text( "I cannot help.", - additional_properties={"model_output_kind": "refusal"}, + additional_properties={"core": {"model_output_kind": "refusal"}}, ) update = create_test_agent_update([content]) diff --git a/python/packages/devui/tests/devui/test_multimodal_workflow.py b/python/packages/devui/tests/devui/test_multimodal_workflow.py index 4699c813c5f..cf272394e66 100644 --- a/python/packages/devui/tests/devui/test_multimodal_workflow.py +++ b/python/packages/devui/tests/devui/test_multimodal_workflow.py @@ -153,14 +153,24 @@ def test_convert_openai_input_preserves_file_ids_and_canonical_media_types(self) result = executor._convert_input_to_chat_message(openai_input) assert result.contents[0].media_type == "image/jpeg" + assert result.contents[0].additional_properties["openai"] == { + "content_type": "input_image", + "image_detail": "low", + } assert result.contents[0].additional_properties["detail"] == "low" assert result.contents[1].type == "hosted_file" assert result.contents[1].file_id == "file_image" + assert result.contents[1].additional_properties["openai"] == { + "content_type": "input_image", + "image_detail": "high", + } assert result.contents[1].additional_properties["openai_content_type"] == "input_image" assert result.contents[1].additional_properties["detail"] == "high" assert result.contents[2].type == "hosted_file" assert result.contents[2].file_id == "file_audio" + assert result.contents[2].additional_properties["openai"]["content_type"] == "input_file" assert result.contents[2].media_type == "audio/mpeg" + assert result.contents[3].additional_properties["openai"]["content_type"] == "input_file" assert result.contents[3].media_type == "audio/mp4" def test_convert_openai_input_rejects_unsupported_only_content(self): diff --git a/python/packages/foundry/agent_framework_foundry/_chat_client.py b/python/packages/foundry/agent_framework_foundry/_chat_client.py index c2d20b188b7..8ef77c8c461 100644 --- a/python/packages/foundry/agent_framework_foundry/_chat_client.py +++ b/python/packages/foundry/agent_framework_foundry/_chat_client.py @@ -15,7 +15,9 @@ FunctionInvocationConfiguration, FunctionInvocationLayer, Message, + get_namespaced_value, load_settings, + with_namespaced_value, ) from agent_framework._compaction import CompactionStrategy, TokenizerProtocol from agent_framework._feature_stage import ExperimentalFeature, experimental @@ -288,7 +290,12 @@ def _prepare_provider_reasoning_item_for_openai( contents: Sequence[Content], ) -> dict[str, Any] | None: for content in reversed(contents): - raw_item = content.additional_properties.get(_FOUNDRY_REASONING_REPLAY_ITEM_KEY) + raw_item = get_namespaced_value( + content.additional_properties, + "foundry", + "reasoning_item", + content.additional_properties.get(_FOUNDRY_REASONING_REPLAY_ITEM_KEY), + ) if isinstance(raw_item, Mapping): payload = self._prepare_foundry_reasoning_replay_item(raw_item) if payload is not None and payload["id"] == reasoning_id: @@ -322,7 +329,14 @@ def _attach_foundry_reasoning_replay_item(cls, contents: list[Content], item: An reasoning_id = cast(str, payload["id"]) for content in reversed(contents): if content.type == "text_reasoning" and content.id == reasoning_id: - content.additional_properties[_FOUNDRY_REASONING_REPLAY_ITEM_KEY] = payload + updated = with_namespaced_value( + content.additional_properties, + "foundry", + "reasoning_item", + payload, + ) + updated[_FOUNDRY_REASONING_REPLAY_ITEM_KEY] = payload + content.additional_properties = updated return contents.append( @@ -330,7 +344,10 @@ def _attach_foundry_reasoning_replay_item(cls, contents: list[Content], item: An id=reasoning_id, text="", raw_representation=item, - additional_properties={_FOUNDRY_REASONING_REPLAY_ITEM_KEY: payload}, + additional_properties={ + "foundry": {"reasoning_item": payload}, + _FOUNDRY_REASONING_REPLAY_ITEM_KEY: payload, + }, ) ) diff --git a/python/packages/foundry/agent_framework_foundry/_oauth_helpers.py b/python/packages/foundry/agent_framework_foundry/_oauth_helpers.py index b60658f0506..1e2fc0124a4 100644 --- a/python/packages/foundry/agent_framework_foundry/_oauth_helpers.py +++ b/python/packages/foundry/agent_framework_foundry/_oauth_helpers.py @@ -78,8 +78,18 @@ def _oauth_consent_contents(raw_item: Any, consent_link: str) -> list[Content]: consent_link = _validate_consent_link(consent_link, item_id) if consent_link: + server_label = getattr(raw_item, "server_label", None) + additional_properties = ( + { + "openai": {"server_label": server_label}, + "server_label": server_label, + } + if isinstance(server_label, str) and server_label + else None + ) content = Content.from_oauth_consent_request( consent_link=consent_link, + additional_properties=additional_properties, raw_representation=raw_item, ) # The consent request is a user-input request: workflows (``AgentExecutor``) register it diff --git a/python/packages/foundry/tests/foundry/test_foundry_chat_client.py b/python/packages/foundry/tests/foundry/test_foundry_chat_client.py index fc2138ea4ac..490fb2cb3ce 100644 --- a/python/packages/foundry/tests/foundry/test_foundry_chat_client.py +++ b/python/packages/foundry/tests/foundry/test_foundry_chat_client.py @@ -656,6 +656,44 @@ def test_streaming_stateless_replay_prefers_completed_foundry_reasoning_item() - } +async def test_stateless_replay_accepts_canonical_foundry_reasoning_metadata() -> None: + project_client = MagicMock() + project_client.get_openai_client.return_value = _make_mock_openai_client() + client = FoundryChatClient(project_client=project_client, model="test-model") + reasoning_item = { + "type": "reasoning", + "id": "rs_canonical", + "response_id": "resp_canonical", + "summary": [], + "content": [], + } + messages = [ + Message( + role="assistant", + contents=[ + Content.from_text_reasoning( + id="rs_canonical", + text="", + additional_properties={"foundry": {"reasoning_item": reasoning_item}}, + ), + Content.from_function_call( + call_id="call_canonical", + name="lookup_probe", + arguments='{"step":1}', + ), + ], + ), + Message( + role="tool", + contents=[Content.from_function_result(call_id="call_canonical", result="probe-result-step-1")], + ), + ] + + _, run_options, _ = await client._prepare_request(messages, {"store": False}) + + assert run_options["input"][0] == reasoning_item + + async def test_stateless_replay_rejects_incomplete_foundry_reasoning_item() -> None: project_client = MagicMock() project_client.get_openai_client.return_value = _make_mock_openai_client() diff --git a/python/packages/foundry/tests/foundry/test_oauth_helpers.py b/python/packages/foundry/tests/foundry/test_oauth_helpers.py index 06e4f4eca7b..292fc4d4552 100644 --- a/python/packages/foundry/tests/foundry/test_oauth_helpers.py +++ b/python/packages/foundry/tests/foundry/test_oauth_helpers.py @@ -71,14 +71,20 @@ def _make_output_item_event( item_type: str = "oauth_consent_request", consent_link: Any = "https://consent.example.com/auth", item_id: str = "oauth-item-1", + server_label: str = "Foundry Toolbox", ) -> MagicMock: - """Create a mock ``response.output_item.added`` event.""" + """Create a mock ``response.output_item.added`` event. + + OAuth consent items use the Responses wire field ``server_label``. First-class + ``server_name`` is the Agent Framework field for function-call content. + """ event = MagicMock() event.type = "response.output_item.added" item = MagicMock() item.type = item_type item.consent_link = consent_link item.id = item_id + item.server_label = server_label event.item = item return event @@ -87,12 +93,14 @@ def _make_top_level_event( *, consent_link: Any = "https://consent.example.com/authorize", event_id: str = "consent-event-1", + server_label: str = "Foundry Toolbox", ) -> MagicMock: """Create a mock ``response.oauth_consent_requested`` event.""" event = MagicMock() event.type = "response.oauth_consent_requested" event.consent_link = consent_link event.id = event_id + event.server_label = server_label return event @@ -121,6 +129,8 @@ def test_parses_output_item_added_with_valid_link() -> None: consent = [c for c in update.contents if c.type == "oauth_consent_request"] assert len(consent) == 1 assert consent[0].consent_link == "https://consent.example.com/auth" + assert consent[0].additional_properties["openai"]["server_label"] == "Foundry Toolbox" + assert consent[0].additional_properties["server_label"] == "Foundry Toolbox" def test_parses_top_level_consent_requested_event() -> None: @@ -132,6 +142,7 @@ def test_parses_top_level_consent_requested_event() -> None: consent = [c for c in update.contents if c.type == "oauth_consent_request"] assert len(consent) == 1 assert consent[0].consent_link == "https://consent.example.com/authorize" + assert consent[0].additional_properties["openai"]["server_label"] == "Foundry Toolbox" def test_empty_contents_for_non_https_link(caplog: pytest.LogCaptureFixture) -> None: diff --git a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py index 1d4d684e8ac..a2d5cf4e1d1 100644 --- a/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py +++ b/python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py @@ -48,6 +48,7 @@ Workflow, WorkflowAgent, add_usage_details, + get_namespaced_value, ) from agent_framework._mcp import _MCP_TOOL_RESULT_HOST_PAYLOAD_KEY # pyright: ignore[reportPrivateUsage] from agent_framework._telemetry import mark_feature_used @@ -227,7 +228,33 @@ def _container_file_citations_from_function_result(content: Content) -> list[_Co def _is_refusal_text_content(content: Content) -> bool: - return content.type == "text" and content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) == _MODEL_OUTPUT_REFUSAL + return ( + content.type == "text" + and get_namespaced_value( + content.additional_properties, + "core", + _MODEL_OUTPUT_KIND_KEY, + content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY), + ) + == _MODEL_OUTPUT_REFUSAL + ) + + +def _refusal_additional_properties() -> dict[str, Any]: + return { + "core": {_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + _MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL, + } + + +def _openai_value(content: Content, key: str, *legacy_keys: str, default: Any = None) -> Any: + value = get_namespaced_value(content.additional_properties, "openai", key, None) + if value is not None: + return value + for legacy_key in legacy_keys: + if legacy_key in content.additional_properties: + return content.additional_properties[legacy_key] + return default def _validate_checkpoint_context_id(context_id: str) -> None: @@ -2380,6 +2407,21 @@ def record_finish_reason(self, finish_reason: str | None) -> None: return self._stream.internal_metadata[_INCOMPLETE_REASON_KEY] = self._incomplete_reason.value + def _consume_text_update(self, text: str, merge_mode: Literal["append", "replace"] | None) -> str: + """Update the authoritative text buffer and return the wire delta still needed.""" + if merge_mode != "replace": + self._accumulated.append(text) + return text + + previous = "".join(self._accumulated) + if text.startswith(previous): + self._accumulated[:] = [text] + return text[len(previous) :] + if not previous: + self._accumulated[:] = [text] + return text + raise RuntimeError("Cannot project a divergent replacement text snapshot onto an append-only Responses stream.") + async def handle_update( self, update: AgentResponseUpdate, @@ -2418,17 +2460,17 @@ async def handle( for event in self._ensure_message_content("refusal", message_id): yield event self._active_message_id = message_id - self._accumulated.append(content.text) - if self._refusal_content is not None: - yield self._refusal_content.emit_delta(content.text) + delta = self._consume_text_update(content.text, content.merge_mode) + if delta and self._refusal_content is not None: + yield self._refusal_content.emit_delta(delta) elif content.type == "text" and content.text is not None: for event in self._ensure_message_content("text", message_id): yield event self._active_message_id = message_id - self._accumulated.append(content.text) - if self._text_content is not None: - yield self._text_content.emit_delta(content.text) + delta = self._consume_text_update(content.text, content.merge_mode) + if delta and self._text_content is not None: + yield self._text_content.emit_delta(delta) elif content.type == "text_reasoning": if self._active_type != "text_reasoning" or (content.id is not None and content.id != self._active_id): @@ -2439,9 +2481,9 @@ async def handle( if encrypted_content := _reasoning_encrypted_content(content): self._reasoning_encrypted_content = encrypted_content if content.text: - self._accumulated.append(content.text) - if self._summary_part is not None: - yield self._summary_part.emit_text_delta(content.text) + delta = self._consume_text_update(content.text, content.merge_mode) + if delta and self._summary_part is not None: + yield self._summary_part.emit_text_delta(delta) elif content.type == "function_call" and content.call_id is not None: # Declaration-only calls replay request metadata after the streamed call. Scope suppression to the @@ -2606,7 +2648,7 @@ async def handle( pending_safety_checks=cast(list[ComputerCallSafetyCheckParam], content.pending_safety_checks or []), status=status, ) - if content.additional_properties.get("computer_action_format") == "single": + if _openai_value(content, "computer_action_format", "computer_action_format") == "single": if len(content.actions) != 1: raise ValueError("A preview computer call requires exactly one action.") item.pop("actions") @@ -2648,7 +2690,7 @@ async def handle( for event in self._close(): yield event function_call: Content = content.function_call # type: ignore - server_label = function_call.additional_properties.get("server_label", "agent_framework") + server_label = function_call.server_name or "agent_framework" request_saved = False async for event in self._stream.output_item_mcp_approval_request( server_label, @@ -2683,7 +2725,12 @@ async def handle( if not _is_allowed_oauth_consent_link(consent_link, self._allowed_oauth_consent_origins): raise ValueError("OAuth consent request content must include an allowed safe HTTPS consent link.") - server_label = content.additional_properties.get("server_label") + server_label = get_namespaced_value( + content.additional_properties, + "openai", + "server_label", + content.additional_properties.get("server_label"), + ) if not isinstance(server_label, str) or not server_label: server_label = getattr(content.raw_representation, "server_label", None) if not isinstance(server_label, str) or not server_label: @@ -3051,13 +3098,24 @@ def _computer_screenshot_from_output(output: Mapping[str, Any]) -> Content: raise ValueError("Computer call output must contain a computer screenshot.") image_url = output.get("image_url") file_id = output.get("file_id") - additional_properties: dict[str, Any] = {} + openai_properties: dict[str, Any] = {} + legacy_properties: dict[str, Any] = {} if detail := output.get("detail"): - additional_properties["detail"] = detail + openai_properties["image_detail"] = detail + legacy_properties["detail"] = detail + additional_properties = { + "openai": openai_properties, + **legacy_properties, + } if isinstance(image_url, str) and image_url: - if file_id is not None: - additional_properties["file_id"] = file_id - return Content.from_uri(image_url, additional_properties=additional_properties) + return Content.from_uri( + image_url, + file_id=file_id if isinstance(file_id, str) else None, + additional_properties={ + **additional_properties, + **({"file_id": file_id} if isinstance(file_id, str) else {}), + }, + ) if isinstance(file_id, str) and file_id: return Content.from_hosted_file(file_id, additional_properties=additional_properties) raise ValueError("Computer screenshot is missing its image URL or file ID.") @@ -3067,13 +3125,13 @@ def _computer_screenshot_to_output(screenshot: Content) -> dict[str, Any]: output: dict[str, Any] = {"type": "computer_screenshot"} if screenshot.type in ("data", "uri") and screenshot.uri: output["image_url"] = screenshot.uri - if file_id := screenshot.additional_properties.get("file_id"): + if file_id := screenshot.file_id or screenshot.additional_properties.get("file_id"): output["file_id"] = file_id elif screenshot.type == "hosted_file" and screenshot.file_id: output["file_id"] = screenshot.file_id else: raise ValueError("A computer screenshot requires image data, a URI, or a hosted file.") - if detail := screenshot.additional_properties.get("detail"): + if detail := _openai_value(screenshot, "image_detail", "detail"): output["detail"] = detail return output @@ -3118,6 +3176,14 @@ async def _item_to_message( return Message(role=item["role"], contents=[_convert_output_message_content(part) for part in item["content"]]) if item["type"] == "function_call": + item_data = cast("Mapping[str, Any]", item) + item_id = item_data.get("id") + status = item_data.get("status") + additional_properties: dict[str, Any] = {} + if isinstance(item_id, str): + additional_properties.update({"openai": {"item_id": item_id}, "fc_id": item_id}) + if status is not None: + additional_properties["status"] = status return Message( role="assistant", contents=[ @@ -3125,6 +3191,8 @@ async def _item_to_message( item["call_id"], item["name"], arguments=item["arguments"], + status=status if isinstance(status, str) else None, + additional_properties=additional_properties, ) ], ) @@ -3133,9 +3201,25 @@ async def _item_to_message( call_id = item.get("call_id") if call_id is None: raise ValueError("Function call output item is missing a call_id.") + item_id = item.get("id") + result_additional_properties = ( + { + "openai": {"item_id": item_id}, + "item_id": item_id, + } + if isinstance(item_id, str) and item_id + else None + ) return Message( role="tool", - contents=[Content.from_function_result(call_id, result=_json_safe_to_str(item["output"]))], + contents=[ + Content.from_function_result( + call_id, + result=_json_safe_to_str(item["output"]), + status=item.get("status"), + additional_properties=result_additional_properties, + ) + ], ) if item["type"] == "reasoning": @@ -3252,6 +3336,7 @@ async def _item_to_message( "file_search", arguments=_json_safe_to_str({"queries": item["queries"]}), informational_only=True, + hosted=True, ) ], ) @@ -3259,7 +3344,7 @@ async def _item_to_message( if item["type"] == "web_search_call": return Message( role="assistant", - contents=[Content.from_function_call(item["id"], "web_search", informational_only=True)], + contents=[Content.from_function_call(item["id"], "web_search", informational_only=True, hosted=True)], ) if item["type"] == "computer_call": @@ -3268,7 +3353,14 @@ async def _item_to_message( actions = plural_actions or ([singular_action] if singular_action is not None else []) if not actions: raise ValueError("Computer call is missing its ordered actions.") - props = {"computer_action_format": "single"} if not plural_actions and singular_action is not None else None + props = ( + { + "openai": {"computer_action_format": "single"}, + "computer_action_format": "single", + } + if not plural_actions and singular_action is not None + else None + ) return Message( role="assistant", contents=[ @@ -3306,6 +3398,7 @@ async def _item_to_message( item["name"], arguments=item["input"], informational_only=True, + hosted=True, ) ], ) @@ -3338,6 +3431,7 @@ async def _item_to_message( "apply_patch", arguments=_json_safe_to_str(item["operation"]), informational_only=True, + hosted=True, ) ], ) @@ -3389,9 +3483,24 @@ async def _output_item_to_message( ValueError: If the OutputItem type is not supported. """ if item["type"] == "oauth_consent_request": + item_data = cast("Mapping[str, Any]", item) + server_label = item_data.get("server_label") + additional_properties = ( + { + "openai": {"server_label": server_label}, + "server_label": server_label, + } + if isinstance(server_label, str) and server_label + else None + ) return Message( role="assistant", - contents=[Content.from_oauth_consent_request(item["consent_link"])], + contents=[ + Content.from_oauth_consent_request( + item["consent_link"], + additional_properties=additional_properties, + ) + ], ) if item["type"] == "structured_outputs": @@ -3421,7 +3530,7 @@ def _convert_output_message_content(content: OutputMessageContent) -> Content: if content["type"] == "refusal": return Content.from_text( content["refusal"], - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + additional_properties=_refusal_additional_properties(), ) # Defensive: `OutputMessageContent` currently only supports `output_text` and `refusal`, @@ -3450,8 +3559,19 @@ def _convert_file_data(data_uri: str, filename: str | None = None) -> Content: else: prefix = f"[File: {filename}]\n" if filename else "" return Content.from_text(f"{prefix}{decoded_text}") - additional_properties = {"filename": filename} if filename else None - return Content.from_uri(data_uri, additional_properties=additional_properties) + additional_properties = ( + { + "openai": {"content_type": "input_file"}, + "openai_content_type": "input_file", + "filename": filename, + } + if filename + else { + "openai": {"content_type": "input_file"}, + "openai_content_type": "input_file", + } + ) + return Content.from_uri(data_uri, name=filename, additional_properties=additional_properties) def _convert_message_content(content: MessageContent) -> Content: @@ -3477,22 +3597,45 @@ def _convert_message_content(content: MessageContent) -> Content: if content["type"] == "refusal": return Content.from_text( content["refusal"], - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + additional_properties=_refusal_additional_properties(), ) if content["type"] == "reasoning_text": return Content.from_text_reasoning(text=content["text"]) if content["type"] == "input_image": + detail = content.get("detail") + additional_properties = { + "openai": { + "content_type": "input_image", + **({"image_detail": detail} if isinstance(detail, str) else {}), + }, + "openai_content_type": "input_image", + **({"detail": detail} if isinstance(detail, str) else {}), + } if image_url := content.get("image_url"): if image_url.startswith("data:"): - return Content.from_uri(image_url) - return Content.from_uri(image_url, media_type="image/*") + return Content.from_uri(image_url, additional_properties=additional_properties) + return Content.from_uri(image_url, media_type="image/*", additional_properties=additional_properties) if file_id := content.get("file_id"): - return Content.from_hosted_file(file_id) + return Content.from_hosted_file(file_id, additional_properties=additional_properties) if content["type"] == "input_file": + filename = content.get("filename") + additional_properties = { + "openai": {"content_type": "input_file"}, + "openai_content_type": "input_file", + **({"filename": filename} if isinstance(filename, str) else {}), + } if file_url := content.get("file_url"): - return Content.from_uri(file_url) + return Content.from_uri( + file_url, + name=filename if isinstance(filename, str) else None, + additional_properties=additional_properties, + ) if file_id := content.get("file_id"): - return Content.from_hosted_file(file_id, name=content.get("filename")) + return Content.from_hosted_file( + file_id, + name=filename if isinstance(filename, str) else None, + additional_properties=additional_properties, + ) if file_data := content.get("file_data"): return _convert_file_data(file_data, content.get("filename")) if content["type"] == "computer_screenshot": diff --git a/python/packages/foundry_hosting/tests/test_responses.py b/python/packages/foundry_hosting/tests/test_responses.py index ae971c82199..71407ce0426 100644 --- a/python/packages/foundry_hosting/tests/test_responses.py +++ b/python/packages/foundry_hosting/tests/test_responses.py @@ -148,7 +148,12 @@ def _make_function_approval_request_content( ) -> Content: """Build a function_approval_request Content with an embedded function_call.""" function_call = Content.from_function_call( - call_id, name, arguments=arguments, additional_properties={"server_label": server_label} + call_id, + name, + arguments=arguments, + hosted=True, + server_name=server_label, + additional_properties={"server_label": server_label}, ) return Content.from_function_approval_request(request_id, function_call) @@ -675,7 +680,10 @@ async def test_item_to_message_marks_refusal_text() -> None: assert message.contents[0].type == "text" assert message.contents[0].text == "I cannot help." - assert message.contents[0].additional_properties == {"model_output_kind": "refusal"} + assert message.contents[0].additional_properties == { + "core": {"model_output_kind": "refusal"}, + "model_output_kind": "refusal", + } class _CapturingASGITransport: @@ -4448,6 +4456,55 @@ async def test_basic_text_streaming(self) -> None: assert len(done_events) == 1 assert done_events[0]["data"]["text"] == "Hello world!" + async def test_text_snapshot_emits_only_missing_delta(self) -> None: + agent = _make_agent( + stream_updates=[ + AgentResponseUpdate( + contents=[Content.from_text("Hello ", merge_mode="append")], + role="assistant", + ), + AgentResponseUpdate( + contents=[Content.from_text("Hello world", merge_mode="replace")], + role="assistant", + ), + ] + ) + + resp = await _post(_make_server(agent), stream=True) + + events = _parse_sse_events(resp.text) + deltas = [event["data"]["delta"] for event in events if event["event"] == "response.output_text.delta"] + done = next(event for event in events if event["event"] == "response.output_text.done") + + assert deltas == ["Hello ", "world"] + assert done["data"]["text"] == "Hello world" + + @pytest.mark.parametrize("replacement", ["Goodbye", "Hello"]) + async def test_divergent_or_shrinking_text_snapshot_fails_stream(self, replacement: str) -> None: + agent = _make_agent( + stream_updates=[ + AgentResponseUpdate( + contents=[Content.from_text("Hello world", merge_mode="append")], + role="assistant", + ), + AgentResponseUpdate( + contents=[Content.from_text(replacement, merge_mode="replace")], + role="assistant", + ), + ] + ) + + resp = await _post(_make_server(agent), stream=True) + + events = _parse_sse_events(resp.text) + deltas = [event["data"]["delta"] for event in events if event["event"] == "response.output_text.delta"] + done = next(event for event in events if event["event"] == "response.output_text.done") + + assert deltas == ["Hello world"] + assert done["data"]["text"] == "Hello world" + assert events[-1]["event"] == "response.failed" + assert "divergent replacement text snapshot" in events[-1]["data"]["response"]["error"]["message"] + async def test_container_file_citation_is_consistent_across_stream_events(self) -> None: resp = await _post( _make_server(_make_agent(stream_updates=_container_file_citation_updates())), @@ -5068,6 +5125,9 @@ async def test_function_call(self) -> None: assert msg.contents[0].type == "function_call" assert msg.contents[0].call_id == "call_1" assert msg.contents[0].name == "get_weather" + assert msg.contents[0].status == "completed" + assert msg.contents[0].additional_properties["openai"]["item_id"] == "fc-1" + assert msg.contents[0].additional_properties["fc_id"] == "fc-1" assert msg.contents[0].informational_only is False async def test_function_call_output(self) -> None: @@ -5348,6 +5408,7 @@ async def test_file_search_call(self) -> None: assert msg.contents[0].name == "file_search" assert '"what is AI"' in (msg.contents[0].arguments or "") assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_web_search_call(self) -> None: from azure.ai.agentserver.responses.models import OutputItemWebSearchToolCall, WebSearchActionSearch @@ -5363,6 +5424,7 @@ async def test_web_search_call(self) -> None: assert msg.contents[0].type == "function_call" assert msg.contents[0].name == "web_search" assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_computer_call(self) -> None: item = cast( @@ -5382,6 +5444,7 @@ async def test_computer_call(self) -> None: assert msg.contents[0].id == "cc-1" assert msg.contents[0].call_id == "call_cc" assert msg.contents[0].actions == [{"type": "click"}] + assert msg.contents[0].additional_properties["openai"]["computer_action_format"] == "single" assert msg.contents[0].additional_properties["computer_action_format"] == "single" assert msg.contents[0].user_input_request is True @@ -5475,6 +5538,7 @@ async def test_custom_tool_call(self) -> None: assert msg.contents[0].name == "my_tool" assert msg.contents[0].arguments == '{"key": "value"}' assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_custom_tool_call_output(self) -> None: item = cast( @@ -5541,6 +5605,7 @@ async def test_apply_patch_call(self) -> None: "diff": "+ new line", } assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_apply_patch_call_output(self) -> None: from azure.ai.agentserver.responses.models import OutputItemApplyPatchToolCallOutput @@ -5570,6 +5635,8 @@ async def test_oauth_consent_request(self) -> None: assert msg.role == "assistant" assert msg.contents[0].type == "oauth_consent_request" assert msg.contents[0].consent_link == "https://example.com/consent" + assert msg.contents[0].additional_properties["openai"]["server_label"] == "my_server" + assert msg.contents[0].additional_properties["server_label"] == "my_server" async def test_structured_outputs_dict(self) -> None: from azure.ai.agentserver.responses.models import StructuredOutputsOutputItem @@ -5686,6 +5753,9 @@ async def test_function_call(self) -> None: assert msg.contents[0].call_id == "call_1" assert msg.contents[0].name == "get_weather" assert msg.contents[0].arguments == '{"city": "NYC"}' + assert msg.contents[0].status == "completed" + assert msg.contents[0].additional_properties["openai"]["item_id"] == "fc-1" + assert msg.contents[0].additional_properties["fc_id"] == "fc-1" assert msg.contents[0].informational_only is False async def test_function_call_output(self) -> None: @@ -5978,6 +6048,7 @@ async def test_file_search_call(self) -> None: assert msg.contents[0].name == "file_search" assert '"what is AI"' in (msg.contents[0].arguments or "") assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_web_search_call(self) -> None: from azure.ai.agentserver.responses.models import ItemWebSearchToolCall @@ -5996,6 +6067,7 @@ async def test_web_search_call(self) -> None: assert msg.contents[0].type == "function_call" assert msg.contents[0].name == "web_search" assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_computer_call(self) -> None: item = cast( @@ -6016,6 +6088,7 @@ async def test_computer_call(self) -> None: assert msg.contents[0].id == "cc-1" assert msg.contents[0].call_id == "call_cc" assert msg.contents[0].actions == [{"type": "click"}] + assert msg.contents[0].additional_properties["openai"]["computer_action_format"] == "single" assert msg.contents[0].additional_properties["computer_action_format"] == "single" assert msg.contents[0].user_input_request is True @@ -6096,6 +6169,7 @@ async def test_custom_tool_call(self) -> None: assert msg.contents[0].name == "my_tool" assert msg.contents[0].arguments == '{"key": "value"}' assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_custom_tool_call_output(self) -> None: from azure.ai.agentserver.responses.models import ItemCustomToolCallOutput @@ -6183,6 +6257,7 @@ async def test_apply_patch_call(self) -> None: "diff": "+ new line", } assert msg.contents[0].informational_only is True + assert msg.contents[0].hosted is True async def test_apply_patch_call_output(self) -> None: from azure.ai.agentserver.responses.models import ApplyPatchToolCallOutputItemParam @@ -7882,7 +7957,7 @@ async def test_recovered_consent_is_tracked_and_not_emitted_twice(self) -> None: duplicate = Content.from_oauth_consent_request( consent_link="https://consent.example.com/obo", - additional_properties={"server_label": "Foundry Toolbox"}, + additional_properties={"openai": {"server_label": "Foundry Toolbox"}}, ) assert [event async for event in tracker.handle(duplicate)] == [] @@ -7921,7 +7996,7 @@ async def test_mid_run_consent_is_persisted_without_an_incomplete_reason(self) - async def test_streaming_mid_run_consent_is_emitted_once_and_can_be_retried(self) -> None: consent = Content.from_oauth_consent_request( consent_link="https://consent.example.com/obo", - additional_properties={"server_label": "Foundry Toolbox"}, + additional_properties={"openai": {"server_label": "Foundry Toolbox"}}, ) async def consent_updates() -> AsyncIterator[AgentResponseUpdate]: diff --git a/python/packages/gemini/agent_framework_gemini/_chat_client.py b/python/packages/gemini/agent_framework_gemini/_chat_client.py index 2f0dceb9528..67ae1b8a506 100644 --- a/python/packages/gemini/agent_framework_gemini/_chat_client.py +++ b/python/packages/gemini/agent_framework_gemini/_chat_client.py @@ -670,19 +670,36 @@ def _convert_message_contents( # No adjacent carrier: fall back to the signature recorded for this call at # parse time. Covers replays where the carrier was dropped in transit. thought_signature = self._recall_thought_signature(call_id) - if ( - content.informational_only - and isinstance(raw_part, types.Part) - and raw_part.tool_call is not None - ): - tool_call = raw_part.tool_call.model_copy( - update={ - "id": call_id, - "args": content.parse_arguments() or {}, - }, - deep=True, + if content.informational_only: + if isinstance(raw_part, types.Part) and raw_part.tool_call is not None: + tool_call = raw_part.tool_call.model_copy( + update={ + "id": call_id, + "args": content.parse_arguments() or {}, + }, + deep=True, + ) + parts.append(raw_part.model_copy(update={"tool_call": tool_call}, deep=True)) + continue + if content.name not in {tool_type.value for tool_type in types.ToolType}: + raise ContentError( + f"Gemini hosted tool-call replay requires a supported tool name, got {content.name!r}." + ) + try: + resolved_tool_type = types.ToolType(content.name or "") + except ValueError as exc: + raise ContentError( + f"Gemini hosted tool-call replay requires a supported tool name, got {content.name!r}." + ) from exc + parts.append( + types.Part( + tool_call=types.ToolCall( + id=call_id, + tool_type=resolved_tool_type, + args=content.parse_arguments() or {}, + ) + ) ) - parts.append(raw_part.model_copy(update={"tool_call": tool_call}, deep=True)) continue if content.name: call_id_to_name[call_id] = content.name @@ -1091,7 +1108,7 @@ def _process_chunk(self, chunk: types.GenerateContentResponse, model: str) -> Ch """ candidate = chunk.candidates[0] if chunk.candidates else None parts: list[types.Part] = (candidate.content.parts or []) if candidate and candidate.content else [] - contents = self._parse_parts(parts) + contents = self._parse_parts(parts, streaming=True) finish_reason = self._map_finish_reason( candidate.finish_reason.name if candidate and candidate.finish_reason else None @@ -1099,7 +1116,7 @@ def _process_chunk(self, chunk: types.GenerateContentResponse, model: str) -> Ch # Attach usage to the final chunk only (when finish_reason is set). if finish_reason and (usage := self._parse_usage(chunk.usage_metadata)): - contents.append(Content.from_usage(usage_details=usage)) + contents.append(Content.from_usage(usage_details=usage, merge_mode="replace")) return ChatResponseUpdate( contents=contents, @@ -1108,11 +1125,12 @@ def _process_chunk(self, chunk: types.GenerateContentResponse, model: str) -> Ch raw_representation=chunk, ) - def _parse_parts(self, parts: Sequence[types.Part]) -> list[Content]: + def _parse_parts(self, parts: Sequence[types.Part], *, streaming: bool = False) -> list[Content]: """Convert Gemini response parts to framework Content objects. Args: parts: Sequence of ``types.Part`` objects from a Gemini response candidate. + streaming: Whether the parts came from a streaming response chunk. Returns: A list of framework ``Content`` objects (text_reasoning, text, function_call, or @@ -1122,10 +1140,22 @@ def _parse_parts(self, parts: Sequence[types.Part]) -> list[Content]: for part in parts: if part.thought: if part.text: - contents.append(Content.from_text_reasoning(text=part.text, raw_representation=part)) + contents.append( + Content.from_text_reasoning( + text=part.text, + merge_mode="append" if streaming else None, + raw_representation=part, + ) + ) continue if part.text is not None: - contents.append(Content.from_text(text=part.text, raw_representation=part)) + contents.append( + Content.from_text( + text=part.text, + merge_mode="append" if streaming else None, + raw_representation=part, + ) + ) elif part.tool_call is not None: tool_call = part.tool_call if tool_call.id: @@ -1143,6 +1173,8 @@ def _parse_parts(self, parts: Sequence[types.Part]) -> list[Content]: name=tool_name, arguments=tool_call.args or {}, informational_only=True, + hosted=True, + merge_mode="replace" if streaming else None, raw_representation=part, ) ) @@ -1167,7 +1199,8 @@ def _parse_parts(self, parts: Sequence[types.Part]) -> list[Content]: if part.thought_signature is not None: contents.append( Content.from_text_reasoning( - protected_data=base64.b64encode(part.thought_signature).decode("utf-8") + protected_data=base64.b64encode(part.thought_signature).decode("utf-8"), + merge_mode="append" if streaming else None, ) ) # Also key it by the resolved call_id so the signature survives surfaces that @@ -1179,6 +1212,7 @@ def _parse_parts(self, parts: Sequence[types.Part]) -> list[Content]: call_id=call_id, name=function_call.name or "", arguments=function_call.args or {}, + merge_mode="replace" if streaming else None, raw_representation=part, ) ) @@ -1193,10 +1227,22 @@ def _parse_parts(self, parts: Sequence[types.Part]) -> list[Content]: ) elif part.executable_code is not None: if part.executable_code.code: - contents.append(Content.from_text(text=part.executable_code.code, raw_representation=part)) + contents.append( + Content.from_text( + text=part.executable_code.code, + merge_mode="append" if streaming else None, + raw_representation=part, + ) + ) elif part.code_execution_result is not None: if part.code_execution_result.output: - contents.append(Content.from_text(text=part.code_execution_result.output, raw_representation=part)) + contents.append( + Content.from_text( + text=part.code_execution_result.output, + merge_mode="append" if streaming else None, + raw_representation=part, + ) + ) else: logger.debug("Skipping unsupported response part from Gemini") return contents diff --git a/python/packages/gemini/tests/test_gemini_client.py b/python/packages/gemini/tests/test_gemini_client.py index 62f50e815a5..6ceeb696011 100644 --- a/python/packages/gemini/tests/test_gemini_client.py +++ b/python/packages/gemini/tests/test_gemini_client.py @@ -17,6 +17,7 @@ ChatClientException, ChatClientInvalidAuthException, ChatClientInvalidRequestException, + ContentError, ) from google.genai import errors as genai_errors from google.genai import types @@ -759,6 +760,12 @@ async def test_unmapped_finish_reason_still_attaches_usage_on_streamed_final_chu updates = [update async for update in stream] assert updates[-1].finish_reason == "TOO_MANY_TOOL_CALLS" + assert all( + content.merge_mode == "append" for update in updates for content in update.contents if content.type == "text" + ) + assert all( + content.merge_mode == "replace" for update in updates for content in update.contents if content.type == "usage" + ) assert any(c.type == "usage" for c in updates[-1].contents) @@ -1285,6 +1292,7 @@ def test_server_side_tool_call_part_is_informational_only() -> None: assert contents[0].name == "FILE_SEARCH" assert contents[0].parse_arguments() == {"query": "docs"} assert contents[0].informational_only is True + assert contents[0].hosted is True def test_server_side_tool_call_part_preserves_raw_part_on_replay() -> None: @@ -1311,6 +1319,38 @@ def test_server_side_tool_call_part_preserves_raw_part_on_replay() -> None: assert parts[0].tool_call.args == {"query": "docs"} +def test_server_side_tool_call_replays_from_first_class_fields_without_raw_part() -> None: + client, _ = _make_gemini_client() + raw_part = types.Part( + tool_call=types.ToolCall(id="search-1", tool_type=types.ToolType.FILE_SEARCH, args={"query": "docs"}) + ) + content = client._parse_parts([raw_part])[0] + content.raw_representation = None + + parts = client._convert_message_contents([content], {}) + + assert len(parts) == 1 + assert parts[0].tool_call is not None + assert parts[0].function_call is None + assert parts[0].tool_call.id == "search-1" + assert parts[0].tool_call.tool_type == types.ToolType.FILE_SEARCH + assert parts[0].tool_call.args == {"query": "docs"} + + +def test_server_side_tool_call_rejects_unsupported_replay_tool_name() -> None: + client, _ = _make_gemini_client() + content = Content.from_function_call( + call_id="search-1", + name="UNKNOWN_SERVER_TOOL", + arguments={"query": "docs"}, + informational_only=True, + hosted=True, + ) + + with pytest.raises(ContentError, match="supported tool name"): + client._convert_message_contents([content], {}) + + def test_server_side_tool_response_part_preserves_raw_part_on_replay() -> None: """Server-side Gemini tool responses stay in their native tool_response representation.""" client, _ = _make_gemini_client() diff --git a/python/packages/github_copilot/agent_framework_github_copilot/_agent.py b/python/packages/github_copilot/agent_framework_github_copilot/_agent.py index 3173be60b88..64a9f125e42 100644 --- a/python/packages/github_copilot/agent_framework_github_copilot/_agent.py +++ b/python/packages/github_copilot/agent_framework_github_copilot/_agent.py @@ -1147,7 +1147,7 @@ def event_handler(event: SessionEvent) -> None: if data.delta_content: update = AgentResponseUpdate( role="assistant", - contents=[Content.from_text(data.delta_content)], + contents=[Content.from_text(data.delta_content, merge_mode="append")], response_id=data.message_id, message_id=data.message_id, raw_representation=event, @@ -1165,7 +1165,13 @@ def event_handler(event: SessionEvent) -> None: model = event.data.model or None if usage_details or finish_reason or model: update = AgentResponseUpdate( - contents=[Content.from_usage(usage_details, raw_representation=event.data)] + contents=[ + Content.from_usage( + usage_details, + merge_mode="replace", + raw_representation=event.data, + ) + ] if usage_details else None, finish_reason=cast(Any, finish_reason), @@ -1181,6 +1187,8 @@ def event_handler(event: SessionEvent) -> None: call_id=tool_call_id, name=tool_name, arguments=arguments, + hosted=True, + merge_mode="replace", raw_representation=event.data, ) update = AgentResponseUpdate( diff --git a/python/packages/github_copilot/tests/test_github_copilot_agent.py b/python/packages/github_copilot/tests/test_github_copilot_agent.py index e32d425fd3f..175398a3d23 100644 --- a/python/packages/github_copilot/tests/test_github_copilot_agent.py +++ b/python/packages/github_copilot/tests/test_github_copilot_agent.py @@ -727,6 +727,7 @@ def mock_on(handler: Any) -> Any: assert isinstance(responses[0], AgentResponseUpdate) assert responses[0].role == "assistant" assert responses[0].contents[0].text == "Hello" + assert responses[0].contents[0].merge_mode == "append" async def test_run_streaming_captures_assistant_usage_event( self, @@ -875,6 +876,7 @@ def mock_on(handler: Any) -> Any: assert content.call_id == "call_abc123" assert content.name == "get_weather" assert content.arguments == {"city": "Seattle"} + assert content.hosted is True assert content.raw_representation is tool_event_data async def test_run_streaming_tool_execution_complete( diff --git a/python/packages/hosting-a2a/AGENTS.md b/python/packages/hosting-a2a/AGENTS.md index 77a34652166..9fd892f8a8c 100644 --- a/python/packages/hosting-a2a/AGENTS.md +++ b/python/packages/hosting-a2a/AGENTS.md @@ -36,6 +36,10 @@ constructs. content-level metadata, while applications own A2A message and artifact boundaries plus message-level metadata. +Inbound A2A message metadata is stored canonically under +`Message.additional_properties["a2a"]` and dual-written to the legacy +`a2a_metadata` key for compatibility. + Card builders return native A2A protobuf values; do not create a parallel card model or subclass. `AgentA2AAdapter` infers built-in Agent Framework skills from the resolved agent's `SkillsProvider` instances by default; `infer_skills=False` diff --git a/python/packages/hosting-a2a/agent_framework_hosting_a2a/_conversion.py b/python/packages/hosting-a2a/agent_framework_hosting_a2a/_conversion.py index 3192dda9f0c..881d14414d4 100644 --- a/python/packages/hosting-a2a/agent_framework_hosting_a2a/_conversion.py +++ b/python/packages/hosting-a2a/agent_framework_hosting_a2a/_conversion.py @@ -240,13 +240,18 @@ def a2a_to_run( if not contents: raise ValueError("A2A message has no supported text, URL, raw, or data parts to convert to a run.") + message_metadata = MessageToDict(message.metadata) if message.metadata else None return AgentRunArgs( messages=[ Message( "user", contents, message_id=message.message_id or None, - additional_properties={"a2a_metadata": MessageToDict(message.metadata)} if message.metadata else None, + additional_properties=( + {"a2a": message_metadata, "a2a_metadata": message_metadata} + if message_metadata is not None + else None + ), raw_representation=message, ) ], diff --git a/python/packages/hosting-a2a/tests/hosting_a2a/test_conversion.py b/python/packages/hosting-a2a/tests/hosting_a2a/test_conversion.py index c9768cefa96..8f5d143158a 100644 --- a/python/packages/hosting-a2a/tests/hosting_a2a/test_conversion.py +++ b/python/packages/hosting-a2a/tests/hosting_a2a/test_conversion.py @@ -70,7 +70,10 @@ def test_a2a_to_run_converts_supported_parts() -> None: assert run["stream"] is True assert run["options"] == {} assert converted.message_id == "message-1" - assert converted.additional_properties == {"a2a_metadata": {"tenant_value": "kept"}} + assert converted.additional_properties == { + "a2a": {"tenant_value": "kept"}, + "a2a_metadata": {"tenant_value": "kept"}, + } assert [content.type for content in converted.contents] == ["text", "uri", "data", "text"] assert converted.contents[0].additional_properties == {"source": "text"} assert converted.contents[1].uri == "https://example.com/image.png" diff --git a/python/packages/hosting-responses/agent_framework_hosting_responses/_parsing.py b/python/packages/hosting-responses/agent_framework_hosting_responses/_parsing.py index 59ae64523aa..0c00c438804 100644 --- a/python/packages/hosting-responses/agent_framework_hosting_responses/_parsing.py +++ b/python/packages/hosting-responses/agent_framework_hosting_responses/_parsing.py @@ -21,7 +21,15 @@ from copy import copy from typing import Any, Literal, cast -from agent_framework import AgentResponse, AgentResponseUpdate, ChatOptions, Content, Message, ResponseStream +from agent_framework import ( + AgentResponse, + AgentResponseUpdate, + ChatOptions, + Content, + Message, + ResponseStream, + get_namespaced_value, +) from agent_framework._telemetry import mark_feature_used from agent_framework_hosting import AgentRunArgs from openai.types.responses import ( @@ -69,13 +77,39 @@ def _is_refusal_text_content(content: Content) -> bool: - return content.type == "text" and content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) == _MODEL_OUTPUT_REFUSAL + return ( + content.type == "text" + and get_namespaced_value( + content.additional_properties, + "core", + _MODEL_OUTPUT_KIND_KEY, + content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY), + ) + == _MODEL_OUTPUT_REFUSAL + ) + + +def _refusal_additional_properties() -> dict[str, Any]: + return { + "core": {_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + _MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL, + } + + +def _openai_value(content: Content, key: str, *legacy_keys: str, default: Any = None) -> Any: + value = get_namespaced_value(content.additional_properties, "openai", key, None) + if value is not None: + return value + for legacy_key in legacy_keys: + if legacy_key in content.additional_properties: + return content.additional_properties[legacy_key] + return default def _code_interpreter_projection_key(content: Content) -> tuple[str, str] | None: if content.type not in {"code_interpreter_tool_call", "code_interpreter_tool_result"}: return None - call_id = content.call_id or content.additional_properties.get("item_id") + call_id = content.call_id or _openai_value(content, "item_id", "item_id") if not isinstance(call_id, str) or not call_id: return None return content.type, call_id @@ -101,7 +135,7 @@ def _content_from_input_item(item: Mapping[str, Any]) -> Content: raise ValueError("refusal requires a non-empty string `refusal`") return Content.from_text( text=refusal, - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + additional_properties=_refusal_additional_properties(), ) if item_type == "input_image": image_url: Any = item.get("image_url") @@ -109,18 +143,49 @@ def _content_from_input_item(item: Mapping[str, Any]) -> Content: image_url = cast("Mapping[str, Any]", image_url).get("url") if not isinstance(image_url, str) or not image_url: raise ValueError("input_image requires a non-empty string `image_url`") - return Content.from_uri(uri=image_url, media_type=None if image_url.startswith("data:image/") else "image/*") + detail = item.get("detail") + return Content.from_uri( + uri=image_url, + media_type=None if image_url.startswith("data:image/") else "image/*", + additional_properties={ + "openai": { + "content_type": "input_image", + **({"image_detail": detail} if isinstance(detail, str) else {}), + }, + "openai_content_type": "input_image", + **({"detail": detail} if isinstance(detail, str) else {}), + }, + ) if item_type == "input_file": file_url = item.get("file_url") if file_url is not None: if not isinstance(file_url, str) or not file_url: raise ValueError("input_file `file_url` must be a non-empty string") - return Content.from_uri(uri=file_url, media_type=item.get("mime_type")) + filename = item.get("filename") + return Content.from_uri( + uri=file_url, + media_type=item.get("mime_type"), + name=filename if isinstance(filename, str) else None, + additional_properties={ + "openai": {"content_type": "input_file"}, + "openai_content_type": "input_file", + **({"filename": filename} if isinstance(filename, str) else {}), + }, + ) file_id = item.get("file_id") if file_id is not None: if not isinstance(file_id, str) or not file_id: raise ValueError("input_file `file_id` must be a non-empty string") - return Content(type="hosted_file", file_id=file_id) + filename = item.get("filename") + return Content.from_hosted_file( + file_id=file_id, + name=filename if isinstance(filename, str) else None, + additional_properties={ + "openai": {"content_type": "input_file"}, + "openai_content_type": "input_file", + **({"filename": filename} if isinstance(filename, str) else {}), + }, + ) raise ValueError("input_file requires a non-empty string `file_url` or `file_id`") raise ValueError(f"Unsupported Responses input content type: {item_type!r}") @@ -764,12 +829,12 @@ def _function_call_output_item(content: Content, *, status: str) -> ResponseOutp return cast( ResponseOutputItem, ResponseFunctionToolCall( - id=content.additional_properties.get("fc_id") if content.additional_properties else None, + id=_openai_value(content, "item_id", "fc_id"), type="function_call", call_id=content.call_id or f"call_{uuid.uuid4().hex}", name=content.name or "tool", arguments=_arguments_to_str(content.arguments), - status=_message_status(status), # type: ignore[arg-type] + status=_message_status(content.status or status), # type: ignore[arg-type] ), ) @@ -794,7 +859,7 @@ def _function_result_output_item(content: Content, *, status: str) -> ResponseOu return cast( ResponseOutputItem, ResponseFunctionToolCallOutputItem( - id=f"fcout_{uuid.uuid4().hex}", + id=_content_item_id(content) or f"fcout_{uuid.uuid4().hex}", type="function_call_output", call_id=content.call_id or f"call_{uuid.uuid4().hex}", output=output, @@ -875,7 +940,7 @@ def _mcp_result_output_item(content: Content, *, status: str) -> ResponseOutputI def _shell_call_output_item(content: Content, *, status: str) -> ResponseOutputItem: return _response_output_item({ - "id": content.additional_properties.get("item_id") or f"shell_{uuid.uuid4().hex}", + "id": _openai_value(content, "item_id", "item_id") or f"shell_{uuid.uuid4().hex}", "type": "shell_call", "call_id": content.call_id or f"call_{uuid.uuid4().hex}", "action": { @@ -899,7 +964,7 @@ def _shell_result_output_item(content: Content, *, status: str) -> ResponseOutpu outputs.append({"stdout": item.stdout or "", "stderr": item.stderr or "", "outcome": outcome}) return _response_output_item({ - "id": content.additional_properties.get("item_id") or f"shellout_{uuid.uuid4().hex}", + "id": _openai_value(content, "item_id", "item_id") or f"shellout_{uuid.uuid4().hex}", "type": "shell_call_output", "call_id": content.call_id or f"call_{uuid.uuid4().hex}", "output": outputs, @@ -913,11 +978,9 @@ def _function_approval_request_output_item(content: Content) -> ResponseOutputIt return _response_output_item({ "id": content.id or f"approval_{uuid.uuid4().hex}", "type": "mcp_approval_request", - "server_label": ( - function_call.additional_properties.get("server_label", "agent_framework") - if function_call is not None - else "agent_framework" - ), + "server_label": function_call.server_name + if function_call is not None and function_call.server_name + else "agent_framework", "name": function_call.name if function_call is not None and function_call.name else "tool", "arguments": _arguments_to_str(function_call.arguments if function_call is not None else None), }) @@ -992,21 +1055,24 @@ def _image_generation_content_result(content: Content) -> str | None: def _content_item_id(content: Content, result_content: Content | None = None) -> str | None: - item_id = content.additional_properties.get("item_id") + item_id = _openai_value(content, "item_id", "item_id") if isinstance(item_id, str) and item_id: return item_id if result_content is not None: - result_item_id = result_content.additional_properties.get("item_id") + result_item_id = _openai_value(result_content, "item_id", "item_id") if isinstance(result_item_id, str) and result_item_id: return result_item_id - return content.call_id or (result_content.call_id if result_content is not None else None) + return None def _content_property(content: Content, result_content: Content | None, key: str) -> Any: - if key in content.additional_properties: - return content.additional_properties[key] - if result_content is not None and key in result_content.additional_properties: - return result_content.additional_properties[key] + value = _openai_value(content, key, key) + if value is not None: + return value + if result_content is not None: + value = _openai_value(result_content, key, key) + if value is not None: + return value return None @@ -1178,6 +1244,21 @@ def part_value(part_type: Literal["output_text", "refusal"], text: str) -> dict[ return {"type": "refusal", "refusal": text} return {"type": "output_text", "text": text, "annotations": []} + def consume_text_update(text: str, merge_mode: Literal["append", "replace"] | None) -> str: + nonlocal content_text + if merge_mode != "replace": + content_text += text + return text + + previous = content_text + if text.startswith(previous): + content_text = text + return text[len(previous) :] + if not previous: + content_text = text + return text + raise RuntimeError("Cannot project a divergent replacement text snapshot onto an append-only Responses stream.") + def close_content_part() -> list[str]: nonlocal content_type, content_text if message_id is None or content_type is None: @@ -1379,8 +1460,8 @@ def apply_streamed_message_ids(response: dict[str, Any]) -> None: for stream_event in close_content_part(): yield stream_event yield open_content_part(requested_content_type) - content_text += text - if not text: + delta = consume_text_update(text, content.merge_mode) + if not delta: continue delta_type = ( "response.refusal.delta" if requested_content_type == "refusal" else "response.output_text.delta" @@ -1389,7 +1470,7 @@ def apply_streamed_message_ids(response: dict[str, Any]) -> None: "item_id": message_id, "output_index": output_index, "content_index": content_index, - "delta": text, + "delta": delta, } if requested_content_type == "output_text": delta_payload["logprobs"] = [] diff --git a/python/packages/hosting-responses/tests/hosting_responses/test_parsing.py b/python/packages/hosting-responses/tests/hosting_responses/test_parsing.py index 6ece3594ac6..21b560b8143 100644 --- a/python/packages/hosting-responses/tests/hosting_responses/test_parsing.py +++ b/python/packages/hosting-responses/tests/hosting_responses/test_parsing.py @@ -111,7 +111,10 @@ def test_message_envelope_marks_refusal_text(self) -> None: assert msgs[0].contents[0].type == "text" assert msgs[0].contents[0].text == "I cannot help." - assert msgs[0].contents[0].additional_properties == {"model_output_kind": "refusal"} + assert msgs[0].contents[0].additional_properties == { + "core": {"model_output_kind": "refusal"}, + "model_output_kind": "refusal", + } def test_message_envelope_rejects_non_object_content_item(self) -> None: with pytest.raises(ValueError, match="content.*object"): @@ -145,8 +148,12 @@ def test_input_file_via_url(self) -> None: assert msgs[0].contents[0].uri == "https://example.com/report.pdf" def test_input_file_via_file_id(self) -> None: - msgs = messages_from_responses_input([{"type": "input_file", "file_id": "file_123"}]) - assert msgs[0].contents[0].file_id == "file_123" + msgs = messages_from_responses_input([{"type": "input_file", "file_id": "file_123", "filename": "report.pdf"}]) + content = msgs[0].contents[0] + assert content.file_id == "file_123" + assert content.name == "report.pdf" + assert content.additional_properties["openai"]["content_type"] == "input_file" + assert content.additional_properties["openai_content_type"] == "input_file" def test_input_file_missing_anchor_raises(self) -> None: with pytest.raises(ValueError, match="input_file"): @@ -162,10 +169,18 @@ def test_pending_text_flushes_before_message_envelope(self) -> None: assert msgs[1].text == "second" def test_image_url_via_string(self) -> None: - msgs = messages_from_responses_input([{"type": "input_image", "image_url": "https://example.com/cat.png"}]) + msgs = messages_from_responses_input([ + {"type": "input_image", "image_url": "https://example.com/cat.png", "detail": "high"} + ]) assert len(msgs) == 1 - # Image content present. - assert any(getattr(c, "uri", None) == "https://example.com/cat.png" for c in msgs[0].contents) + content = msgs[0].contents[0] + assert content.uri == "https://example.com/cat.png" + assert content.additional_properties["openai"] == { + "content_type": "input_image", + "image_detail": "high", + } + assert content.additional_properties["openai_content_type"] == "input_image" + assert content.additional_properties["detail"] == "high" def test_image_url_via_object(self) -> None: msgs = messages_from_responses_input([ @@ -375,9 +390,25 @@ def test_responses_from_run_preserves_tool_role_function_result(self) -> None: assert len(payload["output"]) == 1 assert payload["output"][0]["type"] == "function_call_output" assert payload["output"][0]["call_id"] == "call_1" + assert payload["output"][0]["id"].startswith("fcout_") + assert payload["output"][0]["id"] != "call_1" assert payload["output"][0]["output"] == [{"type": "input_text", "text": "sunny"}] assert payload["output"][0]["status"] == "completed" + def test_responses_from_run_generates_distinct_output_ids_for_reused_call_id(self) -> None: + result = AgentResponse( + messages=[ + Message(role="tool", contents=[Content.from_function_result("reused", result="first")]), + Message(role="tool", contents=[Content.from_function_result("reused", result="second")]), + ] + ) + + payload = responses_from_run(result, response_id="resp_new") + + assert [item["call_id"] for item in payload["output"]] == ["reused", "reused"] + assert all(item["id"].startswith("fcout_") for item in payload["output"]) + assert payload["output"][0]["id"] != payload["output"][1]["id"] + def test_responses_from_run_omits_internal_function_exception(self) -> None: diagnostic = "test-token-value at /srv/private/tool.py" result = AgentResponse( @@ -986,6 +1017,73 @@ async def updates() -> AsyncIterator[AgentResponseUpdate]: content = cast("list[dict[str, object]]", output[0]["content"]) assert content == [{"type": "refusal", "refusal": "I cannot help."}] + async def test_responses_from_streaming_run_consumes_text_snapshot(self) -> None: + async def updates() -> AsyncIterator[AgentResponseUpdate]: + yield AgentResponseUpdate( + contents=[Content.from_text("Hello ", merge_mode="append")], + role="assistant", + ) + yield AgentResponseUpdate( + contents=[Content.from_text("Hello world", merge_mode="replace")], + role="assistant", + ) + + stream = ResponseStream(updates(), finalizer=AgentResponse.from_updates) + + events = [ + event + async for event in responses_from_streaming_run( + stream, + response_id="resp_new", + ) + ] + + deltas = [ + cast(str, _sse_payload(event)["delta"]) + for event in events + if _sse_payload(event)["type"] == "response.output_text.delta" + ] + done = next( + _sse_payload(event) for event in events if _sse_payload(event)["type"] == "response.output_text.done" + ) + + assert deltas == ["Hello ", "world"] + assert done["text"] == "Hello world" + + @pytest.mark.parametrize("replacement", ["Goodbye", "Hello"]) + async def test_responses_from_streaming_run_rejects_divergent_or_shrinking_text_snapshot( + self, replacement: str + ) -> None: + async def updates() -> AsyncIterator[AgentResponseUpdate]: + yield AgentResponseUpdate( + contents=[Content.from_text("Hello world", merge_mode="append")], + role="assistant", + ) + yield AgentResponseUpdate( + contents=[Content.from_text(replacement, merge_mode="replace")], + role="assistant", + ) + + stream = ResponseStream(updates(), finalizer=AgentResponse.from_updates) + + events = [event async for event in responses_from_streaming_run(stream, response_id="resp_new")] + deltas = [ + cast(str, _sse_payload(event)["delta"]) + for event in events + if _sse_payload(event)["type"] == "response.output_text.delta" + ] + done = next( + _sse_payload(event) for event in events if _sse_payload(event)["type"] == "response.output_text.done" + ) + failed = _sse_payload(events[-1]) + response = cast("dict[str, object]", failed["response"]) + error = cast("dict[str, object]", response["error"]) + + assert deltas == ["Hello world"] + assert done["text"] == "Hello world" + assert failed["type"] == "response.failed" + assert "divergent replacement text snapshot" in cast(str, error["message"]) + async def test_responses_from_streaming_run_preserves_mixed_text_and_refusal_parts(self) -> None: marker = {"model_output_kind": "refusal"} diff --git a/python/packages/mistral/agent_framework_mistral/_chat_client.py b/python/packages/mistral/agent_framework_mistral/_chat_client.py index 749c4a437ae..15ebd072158 100644 --- a/python/packages/mistral/agent_framework_mistral/_chat_client.py +++ b/python/packages/mistral/agent_framework_mistral/_chat_client.py @@ -236,7 +236,11 @@ def _tool_call_id_of(tool_call: ToolCall) -> str: return "" -def _function_call_content(tool_call: ToolCall) -> Content: +def _function_call_content( + tool_call: ToolCall, + *, + merge_mode: Literal["append", "replace"] | None = None, +) -> Content: arguments = tool_call.function.arguments if isinstance(arguments, str): normalized_arguments: str | dict[str, Any] = arguments @@ -248,6 +252,7 @@ def _function_call_content(tool_call: ToolCall) -> Content: call_id=_tool_call_id_of(tool_call), name=tool_call.function.name, arguments=normalized_arguments, + merge_mode=merge_mode, raw_representation=tool_call, ) @@ -272,7 +277,7 @@ def add(self, choice_index: int, fragment: ToolCall) -> list[Content]: if pending is not None: fragment_id = _tool_call_id_of(fragment) if fragment_id and (pending_id := _tool_call_id_of(pending)) and pending_id != fragment_id: - flushed.append(_function_call_content(self._pending.pop(key))) + flushed.append(_function_call_content(self._pending.pop(key), merge_mode="replace")) pending = None if pending is None: self._pending[key] = fragment.model_copy(deep=True) @@ -282,10 +287,10 @@ def add(self, choice_index: int, fragment: ToolCall) -> list[Content]: def flush_choice(self, choice_index: int) -> list[Content]: keys = [key for key in self._pending if key[0] == choice_index] - return [_function_call_content(self._pending.pop(key)) for key in keys] + return [_function_call_content(self._pending.pop(key), merge_mode="replace") for key in keys] def flush_all(self) -> list[Content]: - contents = [_function_call_content(pending) for pending in self._pending.values()] + contents = [_function_call_content(pending, merge_mode="replace") for pending in self._pending.values()] self._pending.clear() return contents @@ -815,14 +820,14 @@ def _parse_chunk(self, chunk: CompletionChunk, tool_calls: _StreamedToolCalls) - finish_reason: FinishReason | None = None if chunk.choices: choice = chunk.choices[0] - contents.extend(self._parse_content_chunks(choice.delta)) + contents.extend(self._parse_content_chunks(choice.delta, streaming=True)) for fragment in choice.delta.tool_calls if isinstance(choice.delta.tool_calls, list) else (): contents.extend(tool_calls.add(choice.index, fragment)) if reason := choice.finish_reason: contents.extend(tool_calls.flush_choice(choice.index)) finish_reason = _map_finish_reason(reason) if usage := self._parse_usage(chunk.usage): - contents.append(Content.from_usage(usage_details=usage, raw_representation=chunk)) + contents.append(Content.from_usage(usage_details=usage, merge_mode="replace", raw_representation=chunk)) return ChatResponseUpdate( contents=contents, role="assistant", @@ -840,20 +845,37 @@ def _parse_message_contents(self, message: AssistantMessage) -> list[Content]: contents.extend(_function_call_content(tool_call) for tool_call in message.tool_calls) return contents - def _parse_content_chunks(self, message: AssistantMessage | DeltaMessage) -> list[Content]: + def _parse_content_chunks( + self, + message: AssistantMessage | DeltaMessage, + *, + streaming: bool = False, + ) -> list[Content]: contents: list[Content] = [] content = message.content if isinstance(content, str): if content: - contents.append(Content.from_text(text=content)) + contents.append(Content.from_text(text=content, merge_mode="append" if streaming else None)) elif isinstance(content, list): for chunk in content: if isinstance(chunk, TextChunk): if chunk.text: - contents.append(Content.from_text(text=chunk.text, raw_representation=chunk)) + contents.append( + Content.from_text( + text=chunk.text, + merge_mode="append" if streaming else None, + raw_representation=chunk, + ) + ) elif isinstance(chunk, ThinkChunk): if reasoning := self._thinking_to_text(chunk): - contents.append(Content.from_text_reasoning(text=reasoning, raw_representation=chunk)) + contents.append( + Content.from_text_reasoning( + text=reasoning, + merge_mode="append" if streaming else None, + raw_representation=chunk, + ) + ) else: logger.debug("Skipping unsupported response chunk from Mistral: %s", type(chunk).__name__) return contents @@ -896,7 +918,10 @@ def _parse_usage(self, usage: UsageInfo | None) -> UsageDetails | None: @staticmethod def _parse_response_metadata(usage: UsageInfo | None) -> dict[str, Any] | None: if usage and isinstance(usage.service_tier, str): - return {"service_tier": usage.service_tier} + return { + "mistral": {"service_tier": usage.service_tier}, + "service_tier": usage.service_tier, + } return None # endregion diff --git a/python/packages/mistral/tests/mistral/test_mistral_chat_client.py b/python/packages/mistral/tests/mistral/test_mistral_chat_client.py index 10a1a524b64..ec65d53d396 100644 --- a/python/packages/mistral/tests/mistral/test_mistral_chat_client.py +++ b/python/packages/mistral/tests/mistral/test_mistral_chat_client.py @@ -318,6 +318,7 @@ async def test_get_response_includes_extended_usage() -> None: assert usage_details["mistral.prompt_audio_seconds"] == 4 assert usage_details["mistral.prompt_audio_tokens"] == 12 assert response.additional_properties["service_tier"] == "priority" + assert response.additional_properties["mistral"]["service_tier"] == "priority" @pytest.mark.parametrize("cached_tokens", ["80", 80.5, True, False]) @@ -1110,6 +1111,9 @@ async def test_streaming_response() -> None: stream = client.get_response([Message("user", ["hi"])], stream=True) updates = [update async for update in stream] assert [u.text for u in updates] == ["Hel", "lo", ""] + assert updates[0].contents[0].merge_mode == "append" + assert updates[1].contents[0].merge_mode == "append" + assert updates[2].contents[0].merge_mode == "replace" response = await stream.get_final_response() assert response.text == "Hello" diff --git a/python/packages/ollama/agent_framework_ollama/_chat_client.py b/python/packages/ollama/agent_framework_ollama/_chat_client.py index a7608e3edad..a0bcc046296 100644 --- a/python/packages/ollama/agent_framework_ollama/_chat_client.py +++ b/python/packages/ollama/agent_framework_ollama/_chat_client.py @@ -15,7 +15,7 @@ Sequence, ) from itertools import chain -from typing import Any, ClassVar, Generic, TypedDict +from typing import Any, ClassVar, Generic, Literal, TypedDict from agent_framework import ( BaseChatClient, @@ -587,14 +587,32 @@ def _format_tool_message( messages.append(OllamaMessage(role="tool", content=tool_text, tool_name=tool_name)) return messages - def _parse_contents_from_ollama(self, response: OllamaChatResponse) -> list[Content]: + def _parse_contents_from_ollama( + self, + response: OllamaChatResponse, + *, + streaming: bool = False, + ) -> list[Content]: contents: list[Content] = [] if response.message.thinking: - contents.append(Content.from_text_reasoning(text=response.message.thinking)) + contents.append( + Content.from_text_reasoning( + text=response.message.thinking, + merge_mode="append" if streaming else None, + ) + ) if response.message.content: - contents.append(Content.from_text(text=response.message.content)) + contents.append( + Content.from_text( + text=response.message.content, + merge_mode="append" if streaming else None, + ) + ) if response.message.tool_calls: - tool_calls = self._parse_tool_calls_from_ollama(response.message.tool_calls) + tool_calls = self._parse_tool_calls_from_ollama( + response.message.tool_calls, + merge_mode="replace" if streaming else None, + ) contents.extend(tool_calls) return contents @@ -604,7 +622,7 @@ def _get_finish_reason_from_ollama(self, response: OllamaChatResponse) -> Finish return FinishReason(response.done_reason) if response.done_reason else None def _parse_streaming_response_from_ollama(self, response: OllamaChatResponse) -> ChatResponseUpdate: - contents = self._parse_contents_from_ollama(response) + contents = self._parse_contents_from_ollama(response, streaming=True) finish_reason = None if response.done: usage_details = UsageDetails( @@ -621,7 +639,7 @@ def _parse_streaming_response_from_ollama(self, response: OllamaChatResponse) -> } ) if usage_details: - contents.append(Content.from_usage(usage_details, raw_representation=response)) + contents.append(Content.from_usage(usage_details, merge_mode="replace", raw_representation=response)) finish_reason = self._get_finish_reason_from_ollama(response) return ChatResponseUpdate( contents=contents, @@ -662,7 +680,12 @@ def _parse_response_from_ollama( response_format=response_format, ) - def _parse_tool_calls_from_ollama(self, tool_calls: Sequence[OllamaMessage.ToolCall]) -> list[Content]: + def _parse_tool_calls_from_ollama( + self, + tool_calls: Sequence[OllamaMessage.ToolCall], + *, + merge_mode: Literal["append", "replace"] | None = None, + ) -> list[Content]: resp: list[Content] = [] for tool in tool_calls: name = tool.function.name @@ -674,6 +697,7 @@ def _parse_tool_calls_from_ollama(self, tool_calls: Sequence[OllamaMessage.ToolC call_id=unique_call_id, name=name, arguments=args, + merge_mode=merge_mode, raw_representation=tool.function, ) resp.append(fcc) diff --git a/python/packages/ollama/tests/test_ollama_chat_client.py b/python/packages/ollama/tests/test_ollama_chat_client.py index 4c95a4d0e30..ab185cb3ce1 100644 --- a/python/packages/ollama/tests/test_ollama_chat_client.py +++ b/python/packages/ollama/tests/test_ollama_chat_client.py @@ -479,6 +479,10 @@ async def test_cmc_streaming_maps_done_reason_and_usage( chat_history.append(Message(contents=["hello world"], role="user")) ollama_client = OllamaChatClient() + update = ollama_client._parse_streaming_response_from_ollama(response) + assert next(content for content in update.contents if content.type == "text").merge_mode == "append" + assert next(content for content in update.contents if content.type == "usage").merge_mode == "replace" + result = ollama_client.get_response(messages=chat_history, stream=True) async for _ in result: pass diff --git a/python/packages/openai/agent_framework_openai/_chat_client.py b/python/packages/openai/agent_framework_openai/_chat_client.py index 102a58bd3c9..a5c18ebda85 100644 --- a/python/packages/openai/agent_framework_openai/_chat_client.py +++ b/python/packages/openai/agent_framework_openai/_chat_client.py @@ -74,6 +74,8 @@ UsageDetails, _set_operation_state, # pyright: ignore[reportPrivateUsage] detect_media_type_from_base64, + get_namespaced_value, + merge_additional_properties, validate_tool_mode, ) from agent_framework.exceptions import ( @@ -119,7 +121,9 @@ from ._shared import ( AzureTokenProvider, _attach_prompt_cache_breakpoint, # pyright: ignore[reportPrivateUsage] + _get_openai_metadata, # pyright: ignore[reportPrivateUsage] _is_mp3_media_type, # pyright: ignore[reportPrivateUsage] + _openai_additional_properties, # pyright: ignore[reportPrivateUsage] load_openai_service_settings, maybe_append_azure_endpoint_guidance, ) @@ -160,7 +164,24 @@ class _PromptCacheOptions(TypedDict, total=False): def _is_refusal_text_content(content: Content) -> bool: - return content.type == "text" and content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) == _MODEL_OUTPUT_REFUSAL + return ( + content.type == "text" + and get_namespaced_value( + content.additional_properties, + "core", + _MODEL_OUTPUT_KIND_KEY, + content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY), + ) + == _MODEL_OUTPUT_REFUSAL + ) + + +def _refusal_additional_properties() -> dict[str, Any]: + """Return the generic refusal marker plus its legacy compatibility form.""" + return { + "core": {_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + _MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL, + } DEFAULT_AZURE_OPENAI_RESPONSES_API_VERSION = "preview" @@ -171,10 +192,26 @@ def _is_refusal_text_content(content: Content) -> bool: OPENAI_LOCAL_SHELL_COMMAND_PARTS_KEY = "openai.local_shell_command_parts" OPENAI_SHELL_OUTPUT_TYPE_SHELL_CALL = "shell_call_output" OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL = "local_shell_call_output" +_OPENAI_COMPUTER_ACTION_FORMAT_KEY = "computer_action_format" +_OPENAI_CONTENT_TYPE_KEY = "content_type" +_OPENAI_IMAGE_DETAIL_KEY = "image_detail" +_OPENAI_ITEM_ID_KEY = "item_id" +_OPENAI_LOGPROBS_KEY = "logprobs" +_OPENAI_REASONING_TEXT_KEY = "reasoning_text" +_OPENAI_SHELL_ENVIRONMENT_NAMESPACE_KEY = "shell_environment" +_OPENAI_SHELL_OUTPUT_TYPE_NAMESPACE_KEY = "shell_output_type" +_OPENAI_LOCAL_SHELL_CALL_ITEM_ID_NAMESPACE_KEY = "local_shell_call_item_id" +_OPENAI_LOCAL_SHELL_COMMAND_PARTS_NAMESPACE_KEY = "local_shell_command_parts" _AZURE_AI_SEARCH_CALL_OUTPUT_TYPE = "azure_ai_search_call_output" _AZURE_AI_SEARCH_OUTPUT_EVENT_TYPES = {"response.output_item.added", "response.output_item.done"} _AZURE_AI_SEARCH_OUTPUT_EVENT_PREFIX = "response.azure_ai_search_call_output." + +def _provider_status(value: Any) -> str | None: + """Return a provider status only when the SDK supplied a concrete string.""" + return value if isinstance(value, str) else None + + # Internal marker emitted by `_prepare_content_for_openai` for an # `mcp_server_tool_result` Content. The Responses API expects an `mcp_call` # input item to carry both arguments and output as one item, so result @@ -842,7 +879,6 @@ def _inner_get_response( if stream: function_call_ids: dict[int, tuple[str, str]] = {} - seen_reasoning_delta_item_ids: set[str] = set() seen_function_call_output_ids: set[str] = set() seen_text_annotation_keys: set[tuple[str, int, int]] = set() validated_options: dict[str, Any] | None = None @@ -900,7 +936,6 @@ async def _stream() -> AsyncIterable[ChatResponseUpdate]: chunk, options=validated_options, function_call_ids=function_call_ids, - seen_reasoning_delta_item_ids=seen_reasoning_delta_item_ids, ) if served_model is not None: update.model = served_model @@ -932,7 +967,6 @@ async def _stream() -> AsyncIterable[ChatResponseUpdate]: chunk, options=validated_options, function_call_ids=function_call_ids, - seen_reasoning_delta_item_ids=seen_reasoning_delta_item_ids, ) else: raw_create_response = await client.responses.with_raw_response.create( @@ -946,7 +980,6 @@ async def _stream() -> AsyncIterable[ChatResponseUpdate]: chunk, options=validated_options, function_call_ids=function_call_ids, - seen_reasoning_delta_item_ids=seen_reasoning_delta_item_ids, ) if served_model is not None: update.model = served_model @@ -1018,13 +1051,19 @@ def _finalize_response_updates( self._enrich_mcp_search_citations([content for update in updates for content in update.contents]) response = super()._finalize_response_updates(updates, response_format=response_format) self._mark_completed_computer_calls([content for message in response.messages for content in message.contents]) - logprobs = [ - logprob - for update in updates - for content in update.contents - if content.type == "text" - for logprob in content.additional_properties.get("logprobs", []) - ] + logprobs: list[Any] = [] + for update in updates: + for content in update.contents: + if content.type != "text": + continue + content_logprobs = _get_openai_metadata( + content.additional_properties, + _OPENAI_LOGPROBS_KEY, + "logprobs", + default=[], + ) + if isinstance(content_logprobs, list): + logprobs.extend(cast("list[Any]", content_logprobs)) if logprobs: assistant_text = next( ( @@ -1037,7 +1076,13 @@ def _finalize_response_updates( None, ) if assistant_text is not None: - assistant_text.additional_properties["logprobs"] = logprobs + assistant_text.additional_properties = merge_additional_properties( + assistant_text.additional_properties, + _openai_additional_properties( + {_OPENAI_LOGPROBS_KEY: logprobs}, + legacy={"logprobs": logprobs}, + ), + ) return response @classmethod @@ -1184,7 +1229,11 @@ def _prepare_tools_for_openai( response_tools: list[Any] = [] for tool_item in tools_list: if isinstance(tool_item, FunctionTool) and tool_item.kind == SHELL_TOOL_KIND_VALUE: - shell_env = (tool_item.additional_properties or {}).get(OPENAI_SHELL_ENVIRONMENT_KEY) + shell_env = _get_openai_metadata( + tool_item.additional_properties, + _OPENAI_SHELL_ENVIRONMENT_NAMESPACE_KEY, + OPENAI_SHELL_ENVIRONMENT_KEY, + ) response_tools.append( FunctionShellToolParam( type="shell", @@ -1219,7 +1268,11 @@ def _get_local_shell_tool_name( continue if tool_item.kind != SHELL_TOOL_KIND_VALUE: continue - shell_env = (tool_item.additional_properties or {}).get(OPENAI_SHELL_ENVIRONMENT_KEY) + shell_env = _get_openai_metadata( + tool_item.additional_properties, + _OPENAI_SHELL_ENVIRONMENT_NAMESPACE_KEY, + OPENAI_SHELL_ENVIRONMENT_KEY, + ) if isinstance(shell_env, Mapping) and shell_env.get("type") == "local": # type: ignore[typeddict-item] return tool_item.name return None @@ -1472,8 +1525,11 @@ def get_shell_tool( if base_tool.func is None: raise ValueError("Shell tool requires an executable function.") - additional_properties = dict(base_tool.additional_properties or {}) - additional_properties[OPENAI_SHELL_ENVIRONMENT_KEY] = local_env + additional_properties = _openai_additional_properties( + {_OPENAI_SHELL_ENVIRONMENT_NAMESPACE_KEY: local_env}, + legacy={OPENAI_SHELL_ENVIRONMENT_KEY: local_env}, + ) + additional_properties = merge_additional_properties(base_tool.additional_properties, additional_properties) base_tool.additional_properties = additional_properties base_tool.kind = SHELL_TOOL_KIND_VALUE return base_tool @@ -2068,9 +2124,13 @@ def _prepare_reasoning_items_for_openai( reasoning_texts: list[dict[str, str]] = [] for content in contents: props = content.additional_properties - if status := props.get("status"): + if status := content.status or props.get("status"): item["status"] = status - if reasoning_text_marker := props.get("reasoning_text"): + if reasoning_text_marker := _get_openai_metadata( + props, + _OPENAI_REASONING_TEXT_KEY, + "reasoning_text", + ): reasoning_text = content.text if reasoning_text_marker is True else reasoning_text_marker if isinstance(reasoning_text, str) and reasoning_text: reasoning_texts.append({"type": "reasoning_text", "text": reasoning_text}) @@ -2113,10 +2173,13 @@ def _prepare_content_for_openai( "text": content.text, "annotations": _annotations_to_output_text(getattr(content, "annotations", None)), } - if "logprobs" in content.additional_properties: - output_text["logprobs"] = self._serialize_provider_payload( - content.additional_properties["logprobs"] - ) + logprobs = _get_openai_metadata( + content.additional_properties, + _OPENAI_LOGPROBS_KEY, + "logprobs", + ) + if logprobs is not None: + output_text["logprobs"] = self._serialize_provider_payload(logprobs) return output_text return _attach_prompt_cache_breakpoint( { @@ -2132,9 +2195,13 @@ def _prepare_content_for_openai( props: dict[str, Any] | None = getattr(content, "additional_properties", None) reasoning_text_marker: Any = None if props: - if status := props.get("status"): + if status := content.status or props.get("status"): ret["status"] = status - if reasoning_text_marker := props.get("reasoning_text"): + if reasoning_text_marker := _get_openai_metadata( + props, + _OPENAI_REASONING_TEXT_KEY, + "reasoning_text", + ): reasoning_text = content.text if reasoning_text_marker is True else reasoning_text_marker if isinstance(reasoning_text, str) and reasoning_text: ret["content"] = [{"type": "reasoning_text", "text": reasoning_text}] @@ -2144,9 +2211,13 @@ def _prepare_content_for_openai( ret["summary"].append({"type": "summary_text", "text": content.text}) return ret case "data" | "uri": - openai_content_type = content.additional_properties.get("openai_content_type") + openai_content_type = _get_openai_metadata( + content.additional_properties, + _OPENAI_CONTENT_TYPE_KEY, + "openai_content_type", + ) if openai_content_type == "input_file": - filename = content.additional_properties.get("filename") + filename = content.name or content.additional_properties.get("filename") file_field = "file_url" if content.uri and not content.uri.startswith("data:") else "file_data" file_obj = { "type": "input_file", @@ -2161,11 +2232,14 @@ def _prepare_content_for_openai( result: dict[str, Any] = { "type": "input_image", "image_url": content.uri, - "detail": content.additional_properties.get("detail", "auto") - if content.additional_properties - else "auto", + "detail": _get_openai_metadata( + content.additional_properties, + _OPENAI_IMAGE_DETAIL_KEY, + "detail", + default="auto", + ), } - file_id = content.additional_properties.get("file_id") if content.additional_properties else None + file_id = content.file_id or content.additional_properties.get("file_id") if file_id is not None: result["file_id"] = file_id return _attach_prompt_cache_breakpoint(result, content) @@ -2185,7 +2259,7 @@ def _prepare_content_for_openai( }, } if content.has_top_level_media_type("application"): - filename = getattr(content, "filename", None) or ( + filename = content.name or ( content.additional_properties.get("filename") if hasattr(content, "additional_properties") and content.additional_properties else None @@ -2199,7 +2273,11 @@ def _prepare_content_for_openai( return _attach_prompt_cache_breakpoint(file_obj, content) return {} case "function_call": - shell_output_type = content.additional_properties.get(OPENAI_SHELL_OUTPUT_TYPE_KEY) + shell_output_type = _get_openai_metadata( + content.additional_properties, + _OPENAI_SHELL_OUTPUT_TYPE_NAMESPACE_KEY, + OPENAI_SHELL_OUTPUT_TYPE_KEY, + ) if shell_output_type == OPENAI_SHELL_OUTPUT_TYPE_SHELL_CALL: return self._prepare_shell_transcript_item_for_openai(content, expected_type="shell_call") if shell_output_type == OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL: @@ -2209,7 +2287,11 @@ def _prepare_content_for_openai( return {} fc_id = content.call_id if not replays_local_storage and content.additional_properties: - live_fc_id = content.additional_properties.get("fc_id") + live_fc_id = _get_openai_metadata( + content.additional_properties, + _OPENAI_ITEM_ID_KEY, + "fc_id", + ) if isinstance(live_fc_id, str) and live_fc_id: fc_id = live_fc_id # OpenAI Responses API requires IDs to start with `fc_` @@ -2223,7 +2305,7 @@ def _prepare_content_for_openai( "name": content.name, "arguments": content.arguments, } - if status := content.additional_properties.get("status"): + if status := content.status or content.additional_properties.get("status"): function_call_obj["status"] = status return function_call_obj case "shell_tool_call": @@ -2238,7 +2320,14 @@ def _prepare_content_for_openai( "id": content.id, "call_id": content.call_id, } - if content.additional_properties.get("computer_action_format") == "single": + if ( + _get_openai_metadata( + content.additional_properties, + _OPENAI_COMPUTER_ACTION_FORMAT_KEY, + "computer_action_format", + ) + == "single" + ): if len(content.actions) != 1: raise ChatClientInvalidRequestException("A preview computer call must contain one action.") computer_call_item["action"] = content.actions[0] @@ -2256,7 +2345,7 @@ def _prepare_content_for_openai( screenshot_output: dict[str, Any] = {"type": "computer_screenshot"} if screenshot.type in ("data", "uri") and screenshot.uri: screenshot_output["image_url"] = screenshot.uri - if file_id := screenshot.additional_properties.get("file_id"): + if file_id := screenshot.file_id or screenshot.additional_properties.get("file_id"): screenshot_output["file_id"] = file_id elif screenshot.type == "hosted_file" and screenshot.file_id: screenshot_output["file_id"] = screenshot.file_id @@ -2264,7 +2353,11 @@ def _prepare_content_for_openai( screenshot_output["image_url"] = image_url else: raise ChatClientInvalidRequestException("A computer screenshot must have an image URL or file ID.") - if detail := screenshot.additional_properties.get("detail"): + if detail := _get_openai_metadata( + screenshot.additional_properties, + _OPENAI_IMAGE_DETAIL_KEY, + "detail", + ): screenshot_output["detail"] = detail computer_output_item: dict[str, Any] = { "type": "computer_call_output", @@ -2279,10 +2372,10 @@ def _prepare_content_for_openai( computer_output_item["acknowledged_safety_checks"] = content.acknowledged_safety_checks return computer_output_item case "function_result": - shell_output_type = ( - content.additional_properties.get(OPENAI_SHELL_OUTPUT_TYPE_KEY) - if content.additional_properties - else None + shell_output_type = _get_openai_metadata( + content.additional_properties, + _OPENAI_SHELL_OUTPUT_TYPE_NAMESPACE_KEY, + OPENAI_SHELL_OUTPUT_TYPE_KEY, ) if shell_output_type == OPENAI_SHELL_OUTPUT_TYPE_SHELL_CALL: return { @@ -2290,10 +2383,10 @@ def _prepare_content_for_openai( "type": OPENAI_SHELL_OUTPUT_TYPE_SHELL_CALL, "output": self._to_shell_call_output_payload(content), } - local_shell_call_item_id = ( - content.additional_properties.get(OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY) - if content.additional_properties - else None + local_shell_call_item_id = _get_openai_metadata( + content.additional_properties, + _OPENAI_LOCAL_SHELL_CALL_ITEM_ID_NAMESPACE_KEY, + OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY, ) if shell_output_type == OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL and local_shell_call_item_id: return { @@ -2342,9 +2435,7 @@ def _prepare_content_for_openai( "id": content.id, "arguments": content.function_call.arguments, # type: ignore[union-attr] "name": content.function_call.name, # type: ignore[union-attr] - "server_label": content.function_call.additional_properties.get("server_label") # type: ignore[union-attr] - if content.function_call.additional_properties # type: ignore[union-attr] - else None, + "server_label": content.function_call.server_name, # type: ignore[union-attr] } case "function_approval_response": return { @@ -2378,14 +2469,23 @@ def _prepare_content_for_openai( # the citation context for round-tripping. if role == "assistant": return {} - openai_content_type = content.additional_properties.get("openai_content_type") + openai_content_type = _get_openai_metadata( + content.additional_properties, + _OPENAI_CONTENT_TYPE_KEY, + "openai_content_type", + ) if openai_content_type == "input_image" or ( openai_content_type is None and content.media_type and content.has_top_level_media_type("image") ): return { "type": "input_image", "file_id": content.file_id, - "detail": content.additional_properties.get("detail", "auto"), + "detail": _get_openai_metadata( + content.additional_properties, + _OPENAI_IMAGE_DETAIL_KEY, + "detail", + default="auto", + ), } return { "type": "input_file", @@ -2578,10 +2678,16 @@ def _shell_item_to_contents(self, item: Any, local_shell_tool_name: str | None) call_id=shell_call_id, name=local_shell_tool_name, arguments=json.dumps({"command": "\n".join(shell_commands).strip()}), - additional_properties={ - OPENAI_SHELL_OUTPUT_TYPE_KEY: OPENAI_SHELL_OUTPUT_TYPE_SHELL_CALL, - OPENAI_LOCAL_SHELL_COMMAND_PARTS_KEY: shell_commands, - }, + additional_properties=_openai_additional_properties( + { + _OPENAI_SHELL_OUTPUT_TYPE_NAMESPACE_KEY: OPENAI_SHELL_OUTPUT_TYPE_SHELL_CALL, + _OPENAI_LOCAL_SHELL_COMMAND_PARTS_NAMESPACE_KEY: shell_commands, + }, + legacy={ + OPENAI_SHELL_OUTPUT_TYPE_KEY: OPENAI_SHELL_OUTPUT_TYPE_SHELL_CALL, + OPENAI_LOCAL_SHELL_COMMAND_PARTS_KEY: shell_commands, + }, + ), raw_representation=item, ) ) @@ -2592,7 +2698,7 @@ def _shell_item_to_contents(self, item: Any, local_shell_tool_name: str | None) commands=shell_commands, timeout_ms=shell_timeout_ms, max_output_length=shell_max_output, - status=getattr(item, "status", None), + status=_provider_status(getattr(item, "status", None)), raw_representation=item, ) ) @@ -2616,11 +2722,18 @@ def _shell_item_to_contents(self, item: Any, local_shell_tool_name: str | None) call_id=local_call_id, name=local_shell_tool_name, arguments=json.dumps({"command": local_command}), - additional_properties={ - OPENAI_SHELL_OUTPUT_TYPE_KEY: OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL, - OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY: local_call_item_id, - OPENAI_LOCAL_SHELL_COMMAND_PARTS_KEY: local_command_parts, - }, + additional_properties=_openai_additional_properties( + { + _OPENAI_SHELL_OUTPUT_TYPE_NAMESPACE_KEY: OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL, + _OPENAI_LOCAL_SHELL_CALL_ITEM_ID_NAMESPACE_KEY: local_call_item_id, + _OPENAI_LOCAL_SHELL_COMMAND_PARTS_NAMESPACE_KEY: local_command_parts, + }, + legacy={ + OPENAI_SHELL_OUTPUT_TYPE_KEY: OPENAI_SHELL_OUTPUT_TYPE_LOCAL_SHELL_CALL, + OPENAI_LOCAL_SHELL_CALL_ITEM_ID_KEY: local_call_item_id, + OPENAI_LOCAL_SHELL_COMMAND_PARTS_KEY: local_command_parts, + }, + ), raw_representation=item, ) ) @@ -2630,7 +2743,7 @@ def _shell_item_to_contents(self, item: Any, local_shell_tool_name: str | None) call_id=local_call_id, commands=[local_command] if local_command else [], timeout_ms=getattr(getattr(item, "action", None), "timeout_ms", None), - status=getattr(item, "status", None), + status=_provider_status(getattr(item, "status", None)), raw_representation=item, ) ) @@ -2726,11 +2839,18 @@ def _output_part_content(part: ResponseFunctionCallOutputPart | Mapping[str, Any """ field = partial(_output_part_field, part) detail = field("detail") - extra: dict[str, Any] = {"detail": detail} if detail else {} + openai_values: dict[str, Any] = {_OPENAI_IMAGE_DETAIL_KEY: detail} if detail else {} + legacy_values: dict[str, Any] = {"detail": detail} if detail else {} if (part_type := field("type")) in ("input_image", "input_file"): - extra["openai_content_type"] = part_type + openai_values[_OPENAI_CONTENT_TYPE_KEY] = part_type + legacy_values["openai_content_type"] = part_type if filename := field("filename"): - extra["filename"] = filename + legacy_values["filename"] = filename + extra = ( + _openai_additional_properties(openai_values, legacy=legacy_values) + if openai_values or legacy_values + else None + ) if part_type == "input_text" and isinstance(text := field("text"), str): return Content.from_text(text, raw_representation=part) @@ -2741,7 +2861,8 @@ def _output_part_content(part: ResponseFunctionCallOutputPart | Mapping[str, Any return Content.from_uri( uri, media_type=_media_type_from_uri(uri), - additional_properties=extra or None, + name=filename if isinstance(filename, str) else None, + additional_properties=extra, raw_representation=part, ) @@ -2755,7 +2876,7 @@ def _output_part_content(part: ResponseFunctionCallOutputPart | Mapping[str, Any # know is better left unset than asserted as octet-stream. media_type=_guess_media_type(field("filename"), fallback=None), name=field("filename"), - additional_properties=extra or None, + additional_properties=extra, raw_representation=part, ) @@ -2929,12 +3050,19 @@ def _parse_computer_tool_call_content(self, item: Any) -> Content: call_id = getattr(item, "call_id", None) if not isinstance(item_id, str) or not isinstance(call_id, str): raise ChatClientInvalidRequestException("Computer call is missing its item id or call_id.") - additional_properties = {"computer_action_format": "single"} if is_preview else None + additional_properties = ( + _openai_additional_properties( + {_OPENAI_COMPUTER_ACTION_FORMAT_KEY: "single"}, + legacy={"computer_action_format": "single"}, + ) + if is_preview + else None + ) return Content.from_computer_tool_call( id=item_id, call_id=call_id, actions=parsed_actions, - status=getattr(item, "status", None), + status=_provider_status(getattr(item, "status", None)), pending_safety_checks=self._parse_computer_safety_checks(getattr(item, "pending_safety_checks", None)), additional_properties=additional_properties, raw_representation=item, @@ -2950,15 +3078,34 @@ def _parse_computer_tool_result_content(self, item: Any) -> Content: image_url = output.get("image_url") file_id = output.get("file_id") detail = output.get("detail") - screenshot_properties: dict[str, Any] = {} + openai_screenshot_properties: dict[str, Any] = {} + legacy_screenshot_properties: dict[str, Any] = {} if detail is not None: - screenshot_properties["detail"] = detail + openai_screenshot_properties[_OPENAI_IMAGE_DETAIL_KEY] = detail + legacy_screenshot_properties["detail"] = detail if isinstance(image_url, str) and image_url: - if file_id is not None: - screenshot_properties["file_id"] = file_id - screenshot = Content.from_uri(image_url, additional_properties=screenshot_properties) + screenshot = Content.from_uri( + image_url, + file_id=file_id if isinstance(file_id, str) else None, + additional_properties=_openai_additional_properties( + openai_screenshot_properties, + legacy={ + **legacy_screenshot_properties, + **({"file_id": file_id} if isinstance(file_id, str) else {}), + }, + ), + ) elif isinstance(file_id, str) and file_id: - screenshot = Content.from_hosted_file(file_id=file_id, additional_properties=screenshot_properties) + if isinstance(image_url, str) and image_url: + openai_screenshot_properties["image_url"] = image_url + legacy_screenshot_properties["image_url"] = image_url + screenshot = Content.from_hosted_file( + file_id=file_id, + additional_properties=_openai_additional_properties( + openai_screenshot_properties, + legacy=legacy_screenshot_properties, + ), + ) else: raise ChatClientInvalidRequestException("Computer screenshot is missing its image URL or file ID.") call_id = getattr(item, "call_id", None) @@ -2968,7 +3115,7 @@ def _parse_computer_tool_result_content(self, item: Any) -> Content: id=getattr(item, "id", None), call_id=call_id, screenshot=screenshot, - status=getattr(item, "status", None), + status=_provider_status(getattr(item, "status", None)), acknowledged_safety_checks=self._parse_computer_safety_checks( getattr(item, "acknowledged_safety_checks", None) ), @@ -3185,7 +3332,7 @@ def _parse_search_tool_call_content(self, item: Any) -> Content: call_id=call_id, tool_name=self._get_search_tool_name(item_type), arguments=arguments, - status=getattr(item, "status", None), + status=_provider_status(getattr(item, "status", None)), raw_representation=item, ) @@ -3201,7 +3348,7 @@ def _parse_search_tool_result_content(self, item: Any) -> Content: call_id=call_id, tool_name=self._get_search_tool_name(item_type), result=result, - status=getattr(item, "status", None), + status=_provider_status(getattr(item, "status", None)), raw_representation=item, ) @@ -3213,27 +3360,34 @@ def _parse_hosted_function_call_content( arguments: Any = None, ) -> Content: """Create informational-only function call content for hosted Responses items.""" - additional_properties: dict[str, Any] = {"item_type": item.type} + openai_properties: dict[str, Any] = {"item_type": item.type} + legacy_properties: dict[str, Any] = {"item_type": item.type} match item.type: case "custom_tool_call": call_id = item.call_id if item.id: - additional_properties["item_id"] = item.id + openai_properties[_OPENAI_ITEM_ID_KEY] = item.id + legacy_properties["item_id"] = item.id if item.namespace: - additional_properties["namespace"] = item.namespace + openai_properties["namespace"] = item.namespace + legacy_properties["namespace"] = item.namespace case "tool_search_call": call_id = item.call_id or item.id - additional_properties["item_id"] = item.id - additional_properties["status"] = item.status - additional_properties["execution"] = item.execution + openai_properties[_OPENAI_ITEM_ID_KEY] = item.id + legacy_properties["item_id"] = item.id + openai_properties["execution"] = item.execution + legacy_properties["execution"] = item.execution if item.created_by: - additional_properties["created_by"] = item.created_by + openai_properties["created_by"] = item.created_by + legacy_properties["created_by"] = item.created_by return Content.from_function_call( call_id=call_id, name=name, arguments=self._serialize_provider_payload(arguments), informational_only=True, - additional_properties=additional_properties, + hosted=True, + status=_provider_status(getattr(item, "status", None)), + additional_properties=_openai_additional_properties(openai_properties, legacy=legacy_properties), raw_representation=item, ) @@ -3250,17 +3404,21 @@ def _parse_function_call_output_content(self, item: ResponseFunctionToolCallOutp canonical ``Content`` items by :meth:`_function_call_output_result`, so a returned image or file stays addressable instead of arriving as JSON inside a text item. """ - additional_properties: dict[str, Any] = {"item_type": item.type, "status": item.status} + openai_properties: dict[str, Any] = {"item_type": item.type} + legacy_properties: dict[str, Any] = {"item_type": item.type, "status": item.status} if item.id: - additional_properties["item_id"] = item.id + openai_properties[_OPENAI_ITEM_ID_KEY] = item.id + legacy_properties["item_id"] = item.id # `name` (and the other caller-attribution fields) only exist on newer openai SDKs; the # declared floor of 2.25.0 ships only call_id/id/output/status/type. if tool_name := getattr(item, "name", None): - additional_properties["name"] = tool_name + openai_properties["name"] = tool_name + legacy_properties["name"] = tool_name return Content.from_function_result( call_id=call_id, result=self._function_call_output_result(item.output), - additional_properties=additional_properties, + status=item.status, + additional_properties=_openai_additional_properties(openai_properties, legacy=legacy_properties), raw_representation=item, ) @@ -3324,7 +3482,10 @@ def _parse_response_from_openai( text_content = Content.from_text( text=message_content.text, additional_properties=( - {"logprobs": self._serialize_provider_payload(logprobs)} + _openai_additional_properties( + {_OPENAI_LOGPROBS_KEY: self._serialize_provider_payload(logprobs)}, + legacy={"logprobs": self._serialize_provider_payload(logprobs)}, + ) if logprobs is not None else None ), @@ -3403,7 +3564,7 @@ def _parse_response_from_openai( contents.append( Content.from_text( text=message_content.refusal, - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + additional_properties=_refusal_additional_properties(), raw_representation=message_content, ) ) @@ -3412,14 +3573,21 @@ def _parse_response_from_openai( encrypted_content = getattr(item, "encrypted_content", None) if item_content := getattr(item, "content", None): for index, reasoning_content in enumerate(item_content): - additional_properties: dict[str, Any] = {"reasoning_text": True} + additional_properties = _openai_additional_properties( + {_OPENAI_REASONING_TEXT_KEY: True}, + legacy={"reasoning_text": True}, + ) if hasattr(item, "summary") and item.summary and index < len(item.summary): + additional_properties["openai"]["summary"] = self._serialize_provider_payload( + item.summary[index] + ) additional_properties["summary"] = item.summary[index] contents.append( Content.from_text_reasoning( id=item.id, text=reasoning_content.text, protected_data=encrypted_content if not added_reasoning else None, + status=_provider_status(getattr(item, "status", None)), raw_representation=reasoning_content, additional_properties=additional_properties or None, ) @@ -3432,6 +3600,7 @@ def _parse_response_from_openai( id=item.id, text=summary.text, protected_data=encrypted_content if not added_reasoning else None, + status=_provider_status(getattr(item, "status", None)), raw_representation=summary, ) ) @@ -3445,6 +3614,7 @@ def _parse_response_from_openai( id=item.id, text="", protected_data=encrypted_content, + status=_provider_status(getattr(item, "status", None)), raw_representation=item, additional_properties=additional_properties_empty or None, ) @@ -3474,6 +3644,7 @@ def _parse_response_from_openai( Content.from_code_interpreter_tool_call( call_id=call_id, inputs=[Content.from_text(text=code, raw_representation=item)], + status=_provider_status(getattr(item, "status", None)), raw_representation=item, ) ) @@ -3481,6 +3652,7 @@ def _parse_response_from_openai( Content.from_code_interpreter_tool_result( call_id=call_id, outputs=outputs, + status=_provider_status(getattr(item, "status", None)), raw_representation=item, ) ) @@ -3491,7 +3663,11 @@ def _parse_response_from_openai( name=item.name, arguments=item.arguments, informational_only=item.name == local_shell_tool_name, - additional_properties={"fc_id": item.id, "status": item.status}, + status=item.status, + additional_properties=_openai_additional_properties( + {_OPENAI_ITEM_ID_KEY: item.id}, + legacy={"fc_id": item.id, "status": item.status}, + ), raw_representation=item, ) ) @@ -3528,6 +3704,8 @@ def _parse_response_from_openai( call_id=item.id, name=item.name, arguments=item.arguments, + hosted=True, + server_name=item.server_label, additional_properties={"server_label": item.server_label}, raw_representation=item, ), @@ -3652,6 +3830,7 @@ def _parse_chunk_from_openai( seen_reasoning_delta_item_ids: set[str] | None = None, ) -> ChatResponseUpdate: """Parse an OpenAI Responses API streaming event into a ChatResponseUpdate.""" + del seen_reasoning_delta_item_ids # Read from `options` rather than a parameter; see # `_SEEN_FUNCTION_CALL_OUTPUT_IDS_OPTION` for why the signature must not grow. seen_function_call_output_ids = cast("set[str] | None", options.get(_SEEN_FUNCTION_CALL_OUTPUT_IDS_OPTION)) @@ -3676,7 +3855,10 @@ def output_text_properties(output: Any) -> dict[str, Any] | None: serialized = self._serialize_provider_payload(logprobs) if not isinstance(serialized, list): return None - return {"logprobs": serialized} + return _openai_additional_properties( + {_OPENAI_LOGPROBS_KEY: serialized}, + legacy={"logprobs": serialized}, + ) def parse_annotations( annotations: Sequence[Any], @@ -3768,6 +3950,7 @@ def parse_annotations( contents.append( Content.from_text( text=event_part.text, + merge_mode="append", annotations=annotations or None, additional_properties=output_text_properties(cast(Any, event_part)), raw_representation=event, @@ -3778,7 +3961,8 @@ def parse_annotations( contents.append( Content.from_text( text=event_part.refusal, - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + merge_mode="append", + additional_properties=_refusal_additional_properties(), raw_representation=event, ) ) @@ -3789,11 +3973,29 @@ def parse_annotations( if event_part.type == "output_text": annotations = parse_annotations(event_part.annotations, event.item_id, event.content_index) if annotations: - contents.append(Content.from_text(text="", annotations=annotations, raw_representation=event)) + contents.append( + Content.from_text( + text="", + merge_mode="append", + annotations=annotations, + raw_representation=event, + ) + ) case "response.output_text.delta": contents.append( Content.from_text( text=event.delta, + merge_mode="append", + additional_properties=output_text_properties(cast(Any, event)), + raw_representation=event, + ) + ) + metadata.update(self._get_metadata_from_response(event)) + case "response.output_text.done": + contents.append( + Content.from_text( + text=event.text, + merge_mode="replace", additional_properties=output_text_properties(cast(Any, event)), raw_representation=event, ) @@ -3803,77 +4005,94 @@ def parse_annotations( contents.append( Content.from_text( text=event.delta, - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + merge_mode="append", + additional_properties=_refusal_additional_properties(), + raw_representation=event, + ) + ) + case "response.refusal.done": + contents.append( + Content.from_text( + text=event.refusal, + merge_mode="replace", + additional_properties=_refusal_additional_properties(), raw_representation=event, ) ) case "response.reasoning_text.delta": - if seen_reasoning_delta_item_ids is not None: - seen_reasoning_delta_item_ids.add(event.item_id) contents.append( Content.from_text_reasoning( id=event.item_id, text=event.delta, + merge_mode="append", raw_representation=event, - additional_properties={"reasoning_text": True}, + additional_properties=_openai_additional_properties( + {_OPENAI_REASONING_TEXT_KEY: True}, + legacy={"reasoning_text": True}, + ), ) ) metadata.update(self._get_metadata_from_response(event)) case "response.reasoning_text.done": - # Done event carries the full accumulated text. Emit it only as a - # fallback when no delta was already received for this item_id, to - # avoid duplicating content in downstream accumulators (e.g. ag-ui). - if seen_reasoning_delta_item_ids is None or event.item_id not in seen_reasoning_delta_item_ids: - contents.append( - Content.from_text_reasoning( - id=event.item_id, - text=event.text, - raw_representation=event, - additional_properties={"reasoning_text": True}, - ) + contents.append( + Content.from_text_reasoning( + id=event.item_id, + text=event.text, + merge_mode="replace", + raw_representation=event, + additional_properties=_openai_additional_properties( + {_OPENAI_REASONING_TEXT_KEY: True}, + legacy={"reasoning_text": True}, + ), ) + ) metadata.update(self._get_metadata_from_response(event)) case "response.reasoning_summary_text.delta": - if seen_reasoning_delta_item_ids is not None: - seen_reasoning_delta_item_ids.add(event.item_id) contents.append( Content.from_text_reasoning( id=event.item_id, text=event.delta, + merge_mode="append", raw_representation=event, ) ) metadata.update(self._get_metadata_from_response(event)) case "response.reasoning_summary_text.done": - # Done event carries the full accumulated text. Emit it only as a - # fallback when no delta was already received for this item_id, to - # avoid duplicating content in downstream accumulators (e.g. ag-ui). - if seen_reasoning_delta_item_ids is None or event.item_id not in seen_reasoning_delta_item_ids: - contents.append( - Content.from_text_reasoning( - id=event.item_id, - text=event.text, - raw_representation=event, - ) + contents.append( + Content.from_text_reasoning( + id=event.item_id, + text=event.text, + merge_mode="replace", + raw_representation=event, ) + ) metadata.update(self._get_metadata_from_response(event)) case "response.code_interpreter_call_code.delta": call_id = getattr(event, "call_id", None) or getattr(event, "id", None) or event.item_id - ci_additional_properties = { - "output_index": event.output_index, - "sequence_number": event.sequence_number, - "item_id": event.item_id, - } + ci_additional_properties = _openai_additional_properties( + { + "output_index": event.output_index, + "sequence_number": event.sequence_number, + _OPENAI_ITEM_ID_KEY: event.item_id, + }, + legacy={ + "output_index": event.output_index, + "sequence_number": event.sequence_number, + "item_id": event.item_id, + }, + ) contents.append( Content.from_code_interpreter_tool_call( call_id=call_id, inputs=[ Content.from_text( text=event.delta, + merge_mode="append", raw_representation=event, additional_properties=ci_additional_properties, ) ], + merge_mode="append", raw_representation=event, additional_properties=ci_additional_properties, ) @@ -3884,23 +4103,30 @@ def parse_annotations( # code_interpreter deltas the same way. case "response.code_interpreter_call_code.done": call_id = getattr(event, "call_id", None) or getattr(event, "id", None) or event.item_id - ci_additional_properties = { - "output_index": event.output_index, - "sequence_number": event.sequence_number, - "item_id": event.item_id, - # Authoritative full code; core merges replace prior deltas. - "stream_complete": True, - } + ci_additional_properties = _openai_additional_properties( + { + "output_index": event.output_index, + "sequence_number": event.sequence_number, + _OPENAI_ITEM_ID_KEY: event.item_id, + }, + legacy={ + "output_index": event.output_index, + "sequence_number": event.sequence_number, + "item_id": event.item_id, + }, + ) contents.append( Content.from_code_interpreter_tool_call( call_id=call_id, inputs=[ Content.from_text( text=event.code, + merge_mode="replace", raw_representation=event, additional_properties=ci_additional_properties, ) ], + merge_mode="replace", raw_representation=event, additional_properties=ci_additional_properties, ) @@ -3929,7 +4155,13 @@ def parse_annotations( if event.response.usage: usage = self._parse_usage_from_openai(event.response.usage) if usage: - contents.append(Content.from_usage(usage_details=usage, raw_representation=event)) + contents.append( + Content.from_usage( + usage_details=usage, + merge_mode="replace", + raw_representation=event, + ) + ) case "response.output_item.added": event_item = event.item match event_item.type: @@ -3960,6 +4192,8 @@ def parse_annotations( call_id=event_item.id, name=event_item.name, arguments=event_item.arguments, + hosted=True, + server_name=event_item.server_label, additional_properties={"server_label": event_item.server_label}, raw_representation=event_item, ), @@ -4004,9 +4238,12 @@ def parse_annotations( inputs=[ Content.from_text( text=event_item.code, + merge_mode="append", raw_representation=event_item, ) ], + status=_provider_status(getattr(event_item, "status", None)), + merge_mode="append", raw_representation=event_item, ) ) @@ -4014,6 +4251,8 @@ def parse_annotations( Content.from_code_interpreter_tool_result( call_id=call_id, outputs=outputs, + status=_provider_status(getattr(event_item, "status", None)), + merge_mode="append", raw_representation=event_item, ) ) @@ -4032,18 +4271,26 @@ def parse_annotations( added_reasoning = False if hasattr(event_item, "content") and event_item.content: for index, reasoning_content in enumerate(event_item.content): - additional_properties: dict[str, Any] = {"reasoning_text": True} + additional_properties = _openai_additional_properties( + {_OPENAI_REASONING_TEXT_KEY: True}, + legacy={"reasoning_text": True}, + ) if ( hasattr(event_item, "summary") and event_item.summary and index < len(event_item.summary) ): + additional_properties["openai"]["summary"] = self._serialize_provider_payload( + event_item.summary[index] + ) additional_properties["summary"] = event_item.summary[index] contents.append( Content.from_text_reasoning( id=reasoning_id or None, text=reasoning_content.text, protected_data=getattr(event_item, "encrypted_content", None), + status=_provider_status(getattr(event_item, "status", None)), + merge_mode="append", raw_representation=reasoning_content, additional_properties=additional_properties or None, ) @@ -4057,6 +4304,8 @@ def parse_annotations( id=reasoning_id or None, text="", protected_data=getattr(event_item, "encrypted_content", None), + status=_provider_status(getattr(event_item, "status", None)), + merge_mode="append", raw_representation=event_item, ) ) @@ -4093,10 +4342,40 @@ def parse_annotations( name=name, arguments=event.delta, informational_only=name == local_shell_tool_name, - additional_properties={ - "output_index": event.output_index, - "fc_id": event.item_id, - }, + merge_mode="append", + additional_properties=_openai_additional_properties( + { + "output_index": event.output_index, + _OPENAI_ITEM_ID_KEY: event.item_id, + }, + legacy={ + "output_index": event.output_index, + "fc_id": event.item_id, + }, + ), + raw_representation=event, + ) + ) + case "response.function_call_arguments.done": + call_id, name = function_call_ids.get(event.output_index, (None, None)) + if call_id and name: + contents.append( + Content.from_function_call( + call_id=call_id, + name=name, + arguments=event.arguments, + informational_only=name == local_shell_tool_name, + merge_mode="replace", + additional_properties=_openai_additional_properties( + { + "output_index": event.output_index, + _OPENAI_ITEM_ID_KEY: event.item_id, + }, + legacy={ + "output_index": event.output_index, + "fc_id": event.item_id, + }, + ), raw_representation=event, ) ) @@ -4136,7 +4415,14 @@ def parse_annotations( annotation_index=event.annotation_index, ) if annotations: - contents.append(Content.from_text(text="", annotations=annotations, raw_representation=event)) + contents.append( + Content.from_text( + text="", + merge_mode="append", + annotations=annotations, + raw_representation=event, + ) + ) case "response.output_item.done": done_item = event.item if getattr(done_item, "type", None) == "reasoning": @@ -4147,6 +4433,7 @@ def parse_annotations( id=getattr(done_item, "id", None), text="", protected_data=encrypted_content, + merge_mode="append", raw_representation=done_item, ) ) @@ -4156,7 +4443,12 @@ def parse_annotations( annotations = parse_annotations(part.annotations, done_item.id, content_index) if annotations: contents.append( - Content.from_text(text="", annotations=annotations, raw_representation=event) + Content.from_text( + text="", + merge_mode="append", + annotations=annotations, + raw_representation=event, + ) ) elif getattr(done_item, "type", None) == "mcp_call": call_id = getattr(done_item, "id", None) or getattr(done_item, "call_id", None) or "" diff --git a/python/packages/openai/agent_framework_openai/_chat_completion_client.py b/python/packages/openai/agent_framework_openai/_chat_completion_client.py index 2bed2a57678..92be0f53933 100644 --- a/python/packages/openai/agent_framework_openai/_chat_completion_client.py +++ b/python/packages/openai/agent_framework_openai/_chat_completion_client.py @@ -41,6 +41,7 @@ Message, ResponseStream, UsageDetails, + get_namespaced_value, ) from agent_framework.exceptions import ( ChatClientException, @@ -67,7 +68,9 @@ PROMPT_CACHE_BREAKPOINT_KEY, AzureTokenProvider, _attach_prompt_cache_breakpoint, # pyright: ignore[reportPrivateUsage] + _get_openai_metadata, # pyright: ignore[reportPrivateUsage] _is_mp3_media_type, # pyright: ignore[reportPrivateUsage] + _openai_additional_properties, # pyright: ignore[reportPrivateUsage] load_openai_service_settings, maybe_append_azure_endpoint_guidance, ) @@ -106,7 +109,24 @@ class _PromptCacheOptions(TypedDict, total=False): def _is_refusal_text_content(content: Content) -> bool: - return content.type == "text" and content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY) == _MODEL_OUTPUT_REFUSAL + return ( + content.type == "text" + and get_namespaced_value( + content.additional_properties, + "core", + _MODEL_OUTPUT_KIND_KEY, + content.additional_properties.get(_MODEL_OUTPUT_KIND_KEY), + ) + == _MODEL_OUTPUT_REFUSAL + ) + + +def _refusal_additional_properties() -> dict[str, Any]: + """Return the generic refusal marker plus its legacy compatibility form.""" + return { + "core": {_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + _MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL, + } # Error message shared with tests — extracted to a constant to keep the @@ -919,7 +939,11 @@ def _parse_response_update_from_openai( # See https://github.com/microsoft/agent-framework/issues/3434 if chunk.usage: contents.append( - Content.from_usage(usage_details=self._parse_usage_from_openai(chunk.usage), raw_representation=chunk) + Content.from_usage( + usage_details=self._parse_usage_from_openai(chunk.usage), + merge_mode="replace", + raw_representation=chunk, + ) ) for choice in chunk.choices: @@ -942,7 +966,12 @@ def _parse_response_update_from_openai( if refusal_content := self._parse_refusal_from_openai(choice): choice_contents.append(refusal_content) if reasoning_details := getattr(choice.delta, "reasoning_details", None): - choice_contents.append(Content.from_text_reasoning(protected_data=json.dumps(reasoning_details))) + choice_contents.append( + Content.from_text_reasoning( + protected_data=json.dumps(reasoning_details), + merge_mode="append", + ) + ) if self.response_parser is not None: choice_contents = list(self.response_parser(choice.delta, choice_contents)) contents.extend(choice_contents) @@ -992,7 +1021,11 @@ def _parse_text_from_openai(self, choice: Choice | ChunkChoice) -> Content | Non if message.content: if not isinstance(message.content, str): return None - return Content.from_text(text=message.content, raw_representation=choice) + return Content.from_text( + text=message.content, + merge_mode="append" if isinstance(choice, ChunkChoice) else None, + raw_representation=choice, + ) return None def _parse_refusal_from_openai(self, choice: Choice | ChunkChoice) -> Content | None: @@ -1001,7 +1034,8 @@ def _parse_refusal_from_openai(self, choice: Choice | ChunkChoice) -> Content | if hasattr(message, "refusal") and message.refusal: return Content.from_text( text=message.refusal, - additional_properties={_MODEL_OUTPUT_KIND_KEY: _MODEL_OUTPUT_REFUSAL}, + merge_mode="append" if isinstance(choice, ChunkChoice) else None, + additional_properties=_refusal_additional_properties(), raw_representation=choice, ) return None @@ -1036,6 +1070,7 @@ def _parse_tool_calls_from_openai(self, choice: Choice | ChunkChoice) -> list[Co call_id=tool.id if tool.id else "", name=tool.function.name if tool.function.name else "", arguments=tool.function.arguments if tool.function.arguments else "", + merge_mode="append" if isinstance(choice, ChunkChoice) else None, raw_representation=tool.function, ) # Preserve the streaming tool-call index. Parallel calls interleave @@ -1045,10 +1080,16 @@ def _parse_tool_calls_from_openai(self, choice: Choice | ChunkChoice) -> list[Co # function, not the tool call). tool_index = getattr(tool, "index", None) if tool_index is not None: - fcc.additional_properties["tool_call_index"] = tool_index + openai_properties: dict[str, Any] = {"tool_call_index": tool_index} + legacy_properties: dict[str, Any] = {"tool_call_index": tool_index} choice_index = getattr(choice, "index", None) if choice_index is not None: - fcc.additional_properties["tool_call_choice_index"] = choice_index + openai_properties["tool_call_choice_index"] = choice_index + legacy_properties["tool_call_choice_index"] = choice_index + fcc.additional_properties = _openai_additional_properties( + openai_properties, + legacy=legacy_properties, + ) resp.append(fcc) # When you enable asynchronous content filtering in Azure OpenAI, you may receive empty deltas @@ -1138,8 +1179,10 @@ def _build_openai_messages(self, message: Message) -> list[dict[str, Any]]: } if message.role != "tool" and (author_name := _sanitize_author_name(message.author_name)): args["name"] = author_name - if "reasoning_details" in message.additional_properties and ( - details := message.additional_properties["reasoning_details"] + if details := _get_openai_metadata( + message.additional_properties, + "reasoning_details", + "reasoning_details", ): args["reasoning_details"] = details @@ -1312,7 +1355,11 @@ def _prepare_content_for_openai(self, content: Content) -> dict[str, Any]: } case "data" | "uri" if content.has_top_level_media_type("image"): image_url_obj: dict[str, Any] = {"url": content.uri} - detail = content.additional_properties.get("detail") + detail = _get_openai_metadata( + content.additional_properties, + "image_detail", + "detail", + ) if isinstance(detail, str): image_url_obj["detail"] = detail return _attach_prompt_cache_breakpoint( @@ -1349,7 +1396,7 @@ def _prepare_content_for_openai(self, content: Content) -> dict[str, Any]: ) case "data" | "uri" if content.has_top_level_media_type("application") and content.uri.startswith("data:"): # type: ignore[union-attr] # All application/* media types should be treated as files for OpenAI - filename = getattr(content, "filename", None) or ( + filename = content.name or ( content.additional_properties.get("filename") if hasattr(content, "additional_properties") and content.additional_properties else None diff --git a/python/packages/openai/agent_framework_openai/_shared.py b/python/packages/openai/agent_framework_openai/_shared.py index bb97920cf80..99b87296115 100644 --- a/python/packages/openai/agent_framework_openai/_shared.py +++ b/python/packages/openai/agent_framework_openai/_shared.py @@ -7,6 +7,7 @@ from copy import copy from typing import TYPE_CHECKING, Any, Literal, Union, cast +from agent_framework import get_namespaced_value from agent_framework._settings import SecretString, load_settings from agent_framework._telemetry import APP_INFO, prepend_agent_framework_to_user_agent from agent_framework.exceptions import SettingNotFoundError @@ -55,11 +56,41 @@ PROMPT_CACHE_BREAKPOINT_KEY = "prompt_cache_breakpoint" +OPENAI_ADDITIONAL_PROPS_NAMESPACE = "openai" +_MISSING = object() MP3_MEDIA_TYPES = frozenset({"audio/mp3", "audio/mpeg", "audio/x-mpeg", "audio/mpeg3", "audio/x-mpeg-3"}) +def _get_openai_metadata( # pyright: ignore[reportUnusedFunction] + additional_properties: Mapping[str, Any] | None, + key: str, + *legacy_keys: str, + default: Any = None, +) -> Any: + """Read OpenAI metadata from its namespace, then legacy flat keys.""" + value = get_namespaced_value(additional_properties, OPENAI_ADDITIONAL_PROPS_NAMESPACE, key, _MISSING) + if value is not _MISSING: + return value + for legacy_key in legacy_keys: + if additional_properties and legacy_key in additional_properties: + return additional_properties[legacy_key] + return default + + +def _openai_additional_properties( # pyright: ignore[reportUnusedFunction] + values: Mapping[str, Any], + *, + legacy: Mapping[str, Any] | None = None, +) -> dict[str, Any]: + """Build namespaced OpenAI metadata with optional legacy compatibility keys.""" + return { + OPENAI_ADDITIONAL_PROPS_NAMESPACE: dict(values), + **dict(legacy or {}), + } + + def _is_mp3_media_type( # pyright: ignore[reportUnusedFunction] media_type: str | None, ) -> bool: @@ -87,10 +118,16 @@ def _attach_prompt_cache_breakpoint( # pyright: ignore[reportUnusedFunction] GPT-5.6 and later models accept an explicit cache breakpoint on supported content blocks; users opt in per part via - ``Content.additional_properties["prompt_cache_breakpoint"]``. + ``Content.additional_properties["core"]["prompt_cache_breakpoint"]``. + The legacy top-level key remains readable. """ props = content.additional_properties - breakpoint_value = props.get(PROMPT_CACHE_BREAKPOINT_KEY) if props else None + breakpoint_value = get_namespaced_value( + props, + "core", + PROMPT_CACHE_BREAKPOINT_KEY, + props.get(PROMPT_CACHE_BREAKPOINT_KEY) if props else None, + ) if isinstance(breakpoint_value, Mapping): part[PROMPT_CACHE_BREAKPOINT_KEY] = dict(cast("Mapping[str, Any]", breakpoint_value)) return part diff --git a/python/packages/openai/tests/openai/test_openai_chat_client.py b/python/packages/openai/tests/openai/test_openai_chat_client.py index f55e555d336..15cadc9d9b9 100644 --- a/python/packages/openai/tests/openai/test_openai_chat_client.py +++ b/python/packages/openai/tests/openai/test_openai_chat_client.py @@ -70,7 +70,10 @@ from openai.types.responses.response_reasoning_text_done_event import ( ResponseReasoningTextDoneEvent, ) +from openai.types.responses.response_refusal_delta_event import ResponseRefusalDeltaEvent +from openai.types.responses.response_refusal_done_event import ResponseRefusalDoneEvent from openai.types.responses.response_text_delta_event import ResponseTextDeltaEvent +from openai.types.responses.response_text_done_event import ResponseTextDoneEvent from pydantic import BaseModel from pytest import param @@ -1285,7 +1288,7 @@ def test_streamed_reasoning_text_replays_as_reasoning_content() -> None: response = ChatResponse.from_updates([ client._parse_chunk_from_openai( event, - {}, + {"openai": {"logprobs": []}, "logprobs": []}, function_call_ids, seen_reasoning_delta_item_ids, ) @@ -1391,7 +1394,10 @@ async def test_streamed_reasoning_text_is_stored_once_and_replayed(event_kind: s reasoning = response.messages[0].contents[0] assert reasoning.text == text - assert reasoning.additional_properties == {"reasoning_text": True} + assert reasoning.additional_properties == { + "openai": {"reasoning_text": True}, + "reasoning_text": True, + } await client.get_response( [ @@ -1472,7 +1478,7 @@ def test_streamed_summary_and_reasoning_text_replay_as_one_provider_item() -> No reasoning_contents = response.messages[0].contents assert [(content.text, content.additional_properties) for content in reasoning_contents] == [ ("Visible summary", {}), - ("Private reasoning", {"reasoning_text": True}), + ("Private reasoning", {"openai": {"reasoning_text": True}, "reasoning_text": True}), ] assert reasoning_contents[1].protected_data == "encrypted-mixed" @@ -1653,7 +1659,10 @@ def test_response_content_creation_with_refusal() -> None: assert len(response.messages[0].contents) == 1 assert response.messages[0].contents[0].type == "text" assert response.messages[0].contents[0].text == "I cannot provide that information." - assert response.messages[0].contents[0].additional_properties == {"model_output_kind": "refusal"} + assert response.messages[0].contents[0].additional_properties == { + "core": {"model_output_kind": "refusal"}, + "model_output_kind": "refusal", + } def test_streaming_refusal_delta_creates_marked_text() -> None: @@ -1667,7 +1676,72 @@ def test_streaming_refusal_delta_creates_marked_text() -> None: assert len(update.contents) == 1 assert update.contents[0].type == "text" assert update.contents[0].text == "I cannot help with that." - assert update.contents[0].additional_properties == {"model_output_kind": "refusal"} + assert update.contents[0].additional_properties == { + "core": {"model_output_kind": "refusal"}, + "model_output_kind": "refusal", + } + + +@pytest.mark.parametrize( + ("delta_event", "done_event", "expected_properties"), + [ + ( + ResponseTextDeltaEvent( + type="response.output_text.delta", + content_index=0, + item_id="message_1", + output_index=0, + sequence_number=1, + delta="Hello ", + logprobs=[], + ), + ResponseTextDoneEvent( + type="response.output_text.done", + content_index=0, + item_id="message_1", + output_index=0, + sequence_number=2, + text="Hello world", + logprobs=[], + ), + {"openai": {"logprobs": []}, "logprobs": []}, + ), + ( + ResponseRefusalDeltaEvent( + type="response.refusal.delta", + content_index=0, + item_id="message_2", + output_index=0, + sequence_number=1, + delta="I cannot ", + ), + ResponseRefusalDoneEvent( + type="response.refusal.done", + content_index=0, + item_id="message_2", + output_index=0, + sequence_number=2, + refusal="I cannot help.", + ), + {"core": {"model_output_kind": "refusal"}, "model_output_kind": "refusal"}, + ), + ], +) +def test_streaming_text_done_events_replace_deltas( + delta_event: Any, + done_event: Any, + expected_properties: dict[str, Any], +) -> None: + client = OpenAIChatClient(model="test-model", api_key="test-key") + + updates = [client._parse_chunk_from_openai(event, {}, {}) for event in (delta_event, done_event)] + + assert updates[0].contents[0].merge_mode == "append" + assert updates[1].contents[0].merge_mode == "replace" + assert updates[1].contents[0].additional_properties == expected_properties + response = ChatResponse.from_updates(updates) + expected_text = done_event.text if hasattr(done_event, "text") else done_event.refusal + assert response.messages[0].contents[0].text == expected_text def test_prepare_marked_refusal_text_uses_native_assistant_shape_and_input_text_fallback() -> None: @@ -1858,7 +1932,10 @@ async def test_non_streaming_reasoning_text_is_stored_once_and_replayed() -> Non reasoning = response.messages[0].contents[0] assert reasoning.text == "Private reasoning text" - assert reasoning.additional_properties == {"reasoning_text": True} + assert reasoning.additional_properties == { + "openai": {"reasoning_text": True}, + "reasoning_text": True, + } await client.get_response( [ @@ -7931,8 +8008,8 @@ async def run_streaming(): await run_streaming() -async def test_inner_get_response_streaming_with_response_format_tracks_reasoning_delta_ids() -> None: - """The responses.stream path should suppress reasoning done events after deltas.""" +async def test_inner_get_response_streaming_with_response_format_uses_done_snapshot() -> None: + """The responses.stream path emits a replace snapshot and finalizes without duplication.""" client = OpenAIChatClient(model="test-model", api_key="test-key") messages = [Message(role="user", contents=["Test streaming with format"])] item_id = "reasoning_stream" @@ -7970,7 +8047,9 @@ async def test_inner_get_response_streaming_with_response_format_tracks_reasonin reasoning_chunks = [ content.text for update in updates for content in update.contents if content.type == "text_reasoning" ] - assert reasoning_chunks == ["Hello "] + assert reasoning_chunks == ["Hello ", "Hello "] + response = ChatResponse.from_updates(updates) + assert response.messages[0].contents[0].text == "Hello " def test_prepare_content_for_openai_image_content() -> None: @@ -8217,8 +8296,9 @@ def test_parse_chunk_from_openai_code_interpreter_done() -> None: assert result.contents[0].additional_properties["output_index"] == 0 assert result.contents[0].additional_properties["sequence_number"] == 5 assert result.contents[0].additional_properties["item_id"] == "ci_456" - assert result.contents[0].additional_properties["stream_complete"] is True - assert result.contents[0].inputs[0].additional_properties["stream_complete"] is True + assert result.contents[0].additional_properties["openai"]["item_id"] == "ci_456" + assert result.contents[0].merge_mode == "replace" + assert result.contents[0].inputs[0].merge_mode == "replace" def test_parse_chunk_from_openai_reasoning() -> None: @@ -8295,8 +8375,8 @@ def test_streaming_reasoning_text_delta_event() -> None: mock_metadata.assert_called_once_with(event) -def test_streaming_reasoning_text_done_event_skipped_after_deltas() -> None: - """Test reasoning text done event does not emit content when deltas were already received.""" +def test_streaming_reasoning_text_done_event_replaces_prior_deltas() -> None: + """A reasoning text done event is an explicit authoritative snapshot.""" client = OpenAIChatClient(model="test-model", api_key="test-key") chat_options: dict[str, Any] = {} function_call_ids: dict[int, tuple[str, str]] = {} @@ -8316,7 +8396,11 @@ def test_streaming_reasoning_text_done_event_skipped_after_deltas() -> None: event, chat_options, function_call_ids, seen_reasoning_delta_item_ids ) # type: ignore - assert len(response.contents) == 0 + assert len(response.contents) == 1 + assert response.contents[0].type == "text_reasoning" + assert response.contents[0].id == "reasoning_456" + assert response.contents[0].text == "complete reasoning" + assert response.contents[0].merge_mode == "replace" mock_metadata.assert_called_once_with(event) assert response.additional_properties == {"test": "data"} @@ -8346,6 +8430,7 @@ def test_streaming_reasoning_text_done_event_fallback_without_deltas() -> None: assert response.contents[0].type == "text_reasoning" assert response.contents[0].id == "reasoning_456" assert response.contents[0].text == "complete reasoning" + assert response.contents[0].merge_mode == "replace" mock_metadata.assert_called_once_with(event) assert response.additional_properties == {"test": "data"} @@ -8375,8 +8460,8 @@ def test_streaming_reasoning_summary_text_delta_event() -> None: mock_metadata.assert_called_once_with(event) -def test_streaming_reasoning_summary_text_done_event_skipped_after_deltas() -> None: - """Test reasoning summary text done event does not emit content when deltas were already received.""" +def test_streaming_reasoning_summary_text_done_event_replaces_prior_deltas() -> None: + """A reasoning summary done event is an explicit authoritative snapshot.""" client = OpenAIChatClient(model="test-model", api_key="test-key") chat_options: dict[str, Any] = {} function_call_ids: dict[int, tuple[str, str]] = {} @@ -8396,7 +8481,11 @@ def test_streaming_reasoning_summary_text_done_event_skipped_after_deltas() -> N event, chat_options, function_call_ids, seen_reasoning_delta_item_ids ) # type: ignore - assert len(response.contents) == 0 + assert len(response.contents) == 1 + assert response.contents[0].type == "text_reasoning" + assert response.contents[0].id == "summary_012" + assert response.contents[0].text == "complete summary" + assert response.contents[0].merge_mode == "replace" mock_metadata.assert_called_once_with(event) assert response.additional_properties == {"custom": "meta"} @@ -8426,12 +8515,13 @@ def test_streaming_reasoning_summary_text_done_event_fallback_without_deltas() - assert response.contents[0].type == "text_reasoning" assert response.contents[0].id == "summary_012" assert response.contents[0].text == "complete summary" + assert response.contents[0].merge_mode == "replace" mock_metadata.assert_called_once_with(event) assert response.additional_properties == {"custom": "meta"} def test_streaming_reasoning_deltas_then_done_no_duplication() -> None: - """Sending delta events followed by a done event produces content only from deltas.""" + """A done snapshot replaces prior reasoning deltas during aggregation.""" client = OpenAIChatClient(model="test-model", api_key="test-key") chat_options: dict[str, Any] = {} function_call_ids: dict[int, tuple[str, str]] = {} @@ -8463,25 +8553,29 @@ def test_streaming_reasoning_deltas_then_done_no_duplication() -> None: text="Hello world", ) - all_contents = [] + updates: list[ChatResponseUpdate] = [] with patch.object(client, "_get_metadata_from_response", return_value={}): for event in cast("tuple[Any, ...]", (delta1, delta2, done)): - response = client._parse_chunk_from_openai( + update = client._parse_chunk_from_openai( event, chat_options, function_call_ids, seen_reasoning_delta_item_ids, # type: ignore ) - all_contents.extend(response.contents) + updates.append(update) - assert len(all_contents) == 2 - assert all_contents[0].text == "Hello " - assert all_contents[1].text == "world" - assert "".join(c.text or "" for c in all_contents) == "Hello world" + assert [content.merge_mode for update in updates for content in update.contents] == [ + "append", + "append", + "replace", + ] + response = ChatResponse.from_updates(updates) + assert response.messages[0].contents[0].text == "Hello world" + assert response.messages[0].contents[0].merge_mode is None -async def test_inner_get_response_streaming_create_tracks_reasoning_delta_ids() -> None: - """The responses.create(stream=True) path should suppress reasoning done events after deltas.""" +async def test_inner_get_response_streaming_create_uses_done_snapshot() -> None: + """The responses.create(stream=True) path emits a replace snapshot and finalizes without duplication.""" client = OpenAIChatClient(model="test-model", api_key="test-key") messages = [Message(role="user", contents=["Test streaming"])] item_id = "reasoning_create" @@ -8515,7 +8609,9 @@ async def test_inner_get_response_streaming_create_tracks_reasoning_delta_ids() reasoning_chunks = [ content.text for update in updates for content in update.contents if content.type == "text_reasoning" ] - assert reasoning_chunks == ["Hello "] + assert reasoning_chunks == ["Hello ", "Hello "] + response = ChatResponse.from_updates(updates) + assert response.messages[0].contents[0].text == "Hello " def test_streaming_reasoning_events_preserve_metadata() -> None: diff --git a/python/packages/openai/tests/openai/test_openai_chat_completion_client.py b/python/packages/openai/tests/openai/test_openai_chat_completion_client.py index 92b7e50ead2..7d5fc75b077 100644 --- a/python/packages/openai/tests/openai/test_openai_chat_completion_client.py +++ b/python/packages/openai/tests/openai/test_openai_chat_completion_client.py @@ -1559,7 +1559,10 @@ def test_streaming_chunk_with_refusal(openai_unit_test_env: dict[str, str]) -> N assert len(update.contents) == 1 assert update.contents[0].type == "text" assert update.contents[0].text == "I cannot help." - assert update.contents[0].additional_properties == {"model_output_kind": "refusal"} + assert update.contents[0].additional_properties == { + "core": {"model_output_kind": "refusal"}, + "model_output_kind": "refusal", + } def test_parse_text_with_refusal(openai_unit_test_env: dict[str, str]) -> None: @@ -1596,7 +1599,10 @@ def test_parse_text_with_refusal(openai_unit_test_env: dict[str, str]) -> None: assert len(message.contents) == 1 assert message.contents[0].type == "text" assert message.contents[0].text == "I cannot provide that information." - assert message.contents[0].additional_properties == {"model_output_kind": "refusal"} + assert message.contents[0].additional_properties == { + "core": {"model_output_kind": "refusal"}, + "model_output_kind": "refusal", + } def test_parse_text_and_refusal_preserves_both(openai_unit_test_env: dict[str, str]) -> None: @@ -1626,7 +1632,10 @@ def test_parse_text_and_refusal_preserves_both(openai_unit_test_env: dict[str, s assert [(content.text, content.additional_properties) for content in parsed.messages[0].contents] == [ ("Partial answer.", {}), - ("I cannot continue.", {"model_output_kind": "refusal"}), + ( + "I cannot continue.", + {"core": {"model_output_kind": "refusal"}, "model_output_kind": "refusal"}, + ), ]