Python: DevUI audio input and response playback by fedeoliv · Pull Request #9462 · microsoft/agent-framework

Fernando de Oliveira (fedeoliv) · 2026-10-16T15:48:13Z

Motivation & Context

DevUI can render text conversations and file attachments, but it did not provide a complete path for testing agents that accept audio input or return audio output. This change adds that path so developers can test audio-capable agents directly in DevUI without building a custom UI.

The feature adds browser microphone recording with push-to-talk, preview before send, audio upload handling, audio response playback, optional auto-stop on silence, and clearer audio settings.

Description & Review Guide

What are the major changes?
- Adds microphone recording in DevUI chat with WAV normalization before sending. Supports WAV and MP3 uploads as agent audio input.
- Recording UX: push-to-talk, live mic level feedback, fixed-size recording layout, and optional auto-stop on silence.
- Adds reusable audio preview and playback UI with waveform rendering, scrubber support, removal, download, and optional auto-play for agent audio responses.
- Adds backend mapping for audio request and response content through DevUI conversations, executor handling, and OpenAI-compatible response mapping.
- Adds audio settings under Settings, grouped into Input, Auto-stop on silence, and Output sections.
- Adds tests for audio content mapping, conversation persistence, and multimodal workflow behavior.
- Updates DevUI documentation with user-facing voice input guidance.
What is the impact of these changes?
- Developers can test audio-capable agents from the DevUI chat surface.
- Text-only agents continue to work as before.
- Audio input is sent after the browser finishes a recording or upload. This PR does not add support for the OpenAI Realtime API, realtime audio streaming, or duplex voice sessions.
- DevUI does not transcribe audio. The selected agent or model must handle audio input.
- OpenAI's stateful Conversations schema does not currently model audio input as conversation content, so audio should be routed through an audio-capable agent/backend path rather than relying on OpenAI proxy conversation persistence.
- Microphone recording depends on browser support and requires localhost or HTTPS.
- Uploaded audio input is limited to WAV and MP3.
- Auto-stop on silence is optional because microphone levels vary by browser, device, and room noise.
What do you want reviewers to focus on?
- Whether the OpenAI-compatible audio content mapping matches expected request and response shapes.
- Whether conversation persistence keeps audio content usable across turns without changing text-only behavior.
- Whether the recording and preview UI is accessible, responsive, and clear enough for DevUI users.
- Whether the audio settings are named and grouped at the right level of detail.
- Whether the limitations are clear enough in the README.

Related Issue

Fixes #9359

AI Assistance

No material AI assistance was used.
This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: GitHub Copilot assisted with implementation, testing, and UI refinements.

Contribution Checklist

The code builds clean without any errors or warnings
All unit tests pass, and I have added new tests where possible
The PR follows the Contribution Guidelines
This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
This is not a breaking change. If it is a breaking change, add the breaking change label or add "[BREAKING]" to the title prefix, before or after any language prefix. A workflow keeps the label and title prefix in sync automatically.

Send microphone recordings and uploaded WAV/MP3 files as input_audio content without transcription.

Render agent audio outputs with preview, download, and optional autoplay while preserving DevUI media output items in final responses. Refine microphone startup behavior and audio settings copy for a less noisy chat experience.

Add push-to-talk, live input level, optional silence auto-stop, clearer audio settings, and tighter recording/preview layouts.

Copilot

🟡 Changes recommended

Unresolved recording lifecycle and audio-handling defects can leave microphone capture active, disable the composer, or lose audio content.

6 open findings

Restrict audio data conversion to data URIs · New
Handle top-level output_file audio responses · New
Honor auto-play for OGG and FLAC fallback playback · New
Reject multiple audio file selections · New
Stop microphone stream if permission resolves after unmount · New
Reset isStarting when stopping during microphone warmup · New

What changed in this PR

Adds audio input and response playback to Python DevUI, enabling developers to test audio-capable agents through the existing chat interface.

Changes:

Adds microphone recording, WAV/MP3 uploads, previews, playback controls, and audio settings.
Extends request mapping, response handling, and conversation persistence for audio.
Adds backend tests and user-facing documentation.

File	Description
python/packages/devui/tests/devui/test_multimodal_workflow.py	Tests audio input conversion and format rejection.
python/packages/devui/tests/devui/test_mapper.py	Tests streamed and aggregated audio output.
python/packages/devui/tests/devui/test_conversations.py	Tests audio persistence round trips.
python/packages/devui/README.md	Documents audio features and limitations.
python/packages/devui/frontend/src/types/openai.ts	Adds audio content and auto-play types.
python/packages/devui/frontend/src/types/agent-framework.ts	Adds structured audio input types.
python/packages/devui/frontend/src/stores/devuiStore.ts	Persists audio preferences.
python/packages/devui/frontend/src/hooks/use-audio-recorder.ts	Implements capture, WAV normalization, and analysis.
python/packages/devui/frontend/src/hooks/index.ts	Exports audio utilities.
python/packages/devui/frontend/src/components/ui/file-upload.tsx	Restricts audio selection to WAV/MP3.
python/packages/devui/frontend/src/components/ui/chat-message-input.tsx	Integrates recording and audio attachments.
python/packages/devui/frontend/src/components/ui/audio-message-preview.tsx	Adds recording and playback controls.
python/packages/devui/frontend/src/components/layout/settings-modal.tsx	Exposes audio settings.
python/packages/devui/frontend/src/components/features/agent/message-renderers/OpenAIMessageRenderer.tsx	Passes audio auto-play flags.
python/packages/devui/frontend/src/components/features/agent/message-renderers/OpenAIContentRenderer.tsx	Renders audio previews and players.
python/packages/devui/frontend/src/components/features/agent/agent-view.tsx	Integrates audio into chat messages.
python/packages/devui/agent_framework_devui/models/_openai_custom.py	Adds extension-compatible response models.
python/packages/devui/agent_framework_devui/models/__init__.py	Exports custom response models.
python/packages/devui/agent_framework_devui/_server.py	Serializes extended conversation items.
python/packages/devui/agent_framework_devui/_mapper.py	Preserves audio in mapped responses.
python/packages/devui/agent_framework_devui/_executor.py	Converts structured audio into agent content.
python/packages/devui/agent_framework_devui/_conversations.py	Adds audio conversation serialization.
python/packages/devui/agent_framework_devui/_audio.py	Centralizes audio format helpers.

🧠 Review effort: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Recording limits, audio persistence, and playback behavior still have functional defects.

11 open findings

🧠 Review effort: Balanced

Enhance audio content check to include additional conditions. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Restore the OpenAI response alias to the SDK Response type and remove custom completed/failed event models so the audio backend changes stay localized to DevUI extension output items. Keep typed input_audio parsing for pyright and preserve audio conversion behavior without changing the OpenAI proxy executor contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 259add50-0116-466a-9cbe-3ac3983977d4

Restore the conversation test to assert input_audio round-trip behavior instead of relying on the internal item representation, and keep mapper assertions compatible with the stricter typing checks used in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 259add50-0116-466a-9cbe-3ac3983977d4

Keep native SDK conversation contracts and validate only DevUI media extensions. Share response-media reconciliation and recorder resource ownership while retaining the original audio UI and controls. Add regressions for microphone cancellation, media recovery, generic uploads, and light/dark audio text contrast. Refs microsoft#9359 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 880103e4-e856-451b-b1b2-ddf4b47094b5

Keep chat audio aligned with response text while preserving centered composer previews and existing player styling. Refs microsoft#9359 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 880103e4-e856-451b-b1b2-ddf4b47094b5

Jose Alvarez (jpalvarezl) · 2026-10-22T11:07:20Z

  // For audio files
-  if (isAudio && fileUrl) {
+  if (isAudioFile && fileUrl) {
+    const sourceUrl = content.file_url || fileDataSourceUrl(content.file_data, mimeType) || fileUrl;


This reintroduces the earlier file_data/file_url issue. When both are present, selecting file_url first can lose otherwise valid embedded audio if that URL is expired or inaccessible. Please prefer fileDataSourceUrl(content.file_data, mimeType), fall back to file_url, and add coverage with both fields populated.

thanks Jose Alvarez (@jpalvarezl), good catch! Just fixed it!

Integrate upstream changes while preserving existing DevUI audio behavior. Resolve generated bundle conflicts by rebuilding with the merged frontend dependencies. Refs microsoft#9359

Keep playback and downloads available when valid file_data accompanies an inaccessible file_url. Cover raw base64 and data URI payloads with both fields populated. Refs microsoft#9359

Fernando de Oliveira (fedeoliv) added 7 commits October 8, 2026 15:22

feat(devui): add voice input support

a8c18f0

Send microphone recordings and uploaded WAV/MP3 files as input_audio content without transcription.

feat(devui): add agent audio response support

6984335

Render agent audio outputs with preview, download, and optional autoplay while preserving DevUI media output items in final responses. Refine microphone startup behavior and audio settings copy for a less noisy chat experience.

fix(devui): overlay audio scrubber on waveform

5f0f97f

feat(devui): improve audio recording controls

7d5230d

Add push-to-talk, live input level, optional silence auto-stop, clearer audio settings, and tighter recording/preview layouts.

docs(devui): simplify voice input guidance

3901bb5

docs(devui): mention audio response playback

625f7db

refactor(devui): use OpenAI input audio type

fd2af2b

Copilot AI balanced review requested due to automatic review settings October 16, 2026 15:48

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 16, 2026 15:48 — with GitHub Actions Active

agent-framework-automation Bot added documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python labels Oct 16, 2026

Copilot started reviewing on behalf of Fernando de Oliveira (fedeoliv) October 16, 2026 15:48

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

style(devui): reduce audio formatting noise

630b023

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 16, 2026 17:00 — with GitHub Actions Active

Merge branch 'main' into feature/devui-voice-input

c88942c

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 16, 2026 17:07 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) requested a balanced review from Copilot October 16, 2026 17:07

Copilot started reviewing on behalf of Fernando de Oliveira (fedeoliv) October 16, 2026 17:08

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Refine audio content type validation

9b27221

Enhance audio content check to include additional conditions. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 16, 2026 17:35 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 16, 2026 17:46 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) force-pushed the feature/devui-voice-input branch from b1eed51 to 9b27221 Compare October 16, 2026 17:57

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 16, 2026 17:57 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) added 2 commits October 16, 2026 14:16

fix(devui): address audio review edge cases

3fd46a3

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 16, 2026 19:12 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 20, 2026 21:22 — with GitHub Actions Active

Merge branch 'main' into feature/devui-voice-input

2a9bb3a

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 20, 2026 21:43 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 20, 2026 22:09 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 20, 2026 22:10 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 20, 2026 22:12 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) marked this pull request as ready for review October 20, 2026 22:22

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 20, 2026 22:22 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 21, 2026 12:01 — with GitHub Actions Active

Jose Alvarez (jpalvarezl) reviewed Oct 22, 2026

View reviewed changes

Fernando de Oliveira (fedeoliv) added 2 commits October 22, 2026 13:28

chore(devui): merge main and rebuild UI assets

8177cf3

Integrate upstream changes while preserving existing DevUI audio behavior. Resolve generated bundle conflicts by rebuilding with the merged frontend dependencies. Refs microsoft#9359

fix(devui): prefer embedded audio over file URLs

e934dc4

Keep playback and downloads available when valid file_data accompanies an inaccessible file_url. Cover raw base64 and data URI payloads with both fields populated. Refs microsoft#9359

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 22, 2026 19:20 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) added 2 commits October 22, 2026 15:59

chore(devui): merge main and resolve UI bundle conflict

4a950b4

refactor(devui): remove redundant audio URL fallback

0e7748f

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 22, 2026 20:11 — with GitHub Actions Active

Merge branch 'main' into feature/devui-voice-input

562cd5e

Fernando de Oliveira (fedeoliv) deployed to github-app-auth October 23, 2026 12:05 — with GitHub Actions Active

Fernando de Oliveira (fedeoliv) requested a review from Jose Alvarez (jpalvarezl) October 23, 2026 12:26

Navigation Menu

Repository navigation

Python: DevUI audio input and response playback - #9462

Python: DevUI audio input and response playback#9462
Fernando de Oliveira (fedeoliv) wants to merge 24 commits into
microsoft:mainfrom
fedeoliv:feature/devui-voice-input

Fernando de Oliveira (fedeoliv) commented Oct 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jose Alvarez (jpalvarezl) Oct 22, 2026

Uh oh!

Fernando de Oliveira (fedeoliv) Oct 22, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Fernando de Oliveira (fedeoliv) commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation & Context

Description & Review Guide

Related Issue

AI Assistance

Contribution Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jose Alvarez (jpalvarezl) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Fernando de Oliveira (fedeoliv) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fernando de Oliveira (fedeoliv) commented Oct 16, 2026 •

edited

Loading