Repository navigation
feat: add native audio input encoding to provider formats - #77
Conversation
PsiACE
changed the title
PsiACE
changed the title
dde0fcd to
449debd
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: needs changes — a .wav/.aiff file cannot reach Gemini with the inferred media type.
Loading audio from a path, a URL, or a WAV/AIFF data URL yields the legacy aliases audio/x-wav / audio/x-aiff, which the Chat Completions encoder canonicalizes (audio/x-wav → wav) but the Gemini encoder forwards verbatim. The documented Google example (republic.audio("path/to/audio.wav") with google:MODEL_ID) therefore sends a MIME type outside Gemini's supported audio set. Evidence and reproduction are in the comment on src/republic/formats/gemini.py.
A separate, low-impact note on the _guess_media_type change is attached to src/republic/_content.py.
| return {"role": "user", "parts": [_function_response(result) for result in message.tool_results]} | ||
| parts: list[Mapping[str, Any]] = provider_payloads(message, GeminiFormat.name) | ||
| parts.extend(_part(part) for part in message.parts if isinstance(part, Text | Image | Video)) | ||
| parts.extend(_part(part) for part in message.parts if isinstance(part, Text | Image | Audio | Video)) |
There was a problem hiding this comment.
Audio is admitted for Gemini here, but _part() forwards part.media_type verbatim to inlineData/fileData mimeType, and the type Republic infers for .wav/.aiff is one Gemini does not accept.
mimetypesmaps.wav→audio/x-wavand.aif/.aiff→audio/x-aiff. That is CPython's built-in table (mimetypes.init(files=[])returns the same), so it is not a property of this machine.- Gemini's accepted audio types are
audio/wav,audio/mp3,audio/aiff,audio/aac,audio/ogg,audio/flac,audio/mpeg,audio/m4a,audio/l16,audio/opus,audio/alaw,audio/mulaw,audio/webm— the "Supported audio formats" list in the Gemini audio guide, themime_typeenum in the Interactions API reference, and Vertex's per-model MIME table agree, and none of them lists thex-aliases. - Chat Completions treats the same alias as supported (
ChatFormat.AUDIO_FORMATSmapsaudio/x-wav→wav), so an identical call succeeds on OpenAI-compatible providers and fails on Gemini, including the example added atdocs/reference/provider-api.md:102.
Reproduction (candidate 449debd, repository test double)
import republic
from tests.conftest import FakeService
service = FakeService()
service.reply_json({"candidates": [{"content": {"parts": [{"text": "heard"}]}, "finishReason": "STOP"}]})
model = republic.get_model("google:test", http_client=service.client())
await model.chat(republic.user("Listen", republic.audio("voice.wav"))) # real .wav file on disk
service.body()["contents"][0]["parts"]
# [{'text': 'Listen'}, {'inlineData': {'mimeType': 'audio/x-wav', 'data': 'UklGRi4uLi5XQVZFZm10IA=='}}]
# the same file through the chat encoder
# [{'type': 'text', 'text': 'Listen'}, {'type': 'input_audio', 'input_audio': {'data': '...', 'format': 'wav'}}]No live model call was made (the review environment has no credentials); what is verified is the encoded request body, the stdlib inference, and the provider's documented MIME set.
Repair direction: canonicalize the legacy aliases before Gemini encoding (audio/x-wav → audio/wav, audio/x-aiff → audio/aiff), in the Gemini encoder or in the media loader, and cover it with a Gemini case that loads a .wav by path — the new Gemini tests always pass media_type= explicitly, so they never exercise the inferred alias. If pass-through is intended instead, the Google example needs media_type="audio/wav" plus a note that Gemini wants canonical MIME types.
|
|
||
| def _guess_media_type(media_class: type[_Media], name: str) -> str: | ||
| guessed, _ = mimetypes.guess_type(name) | ||
| path = urlsplit(name).path if name.startswith(_REMOTE_PREFIXES) else name |
There was a problem hiding this comment.
Minor, non-blocking: this extraction is a no-op on the supported Pythons, and its only observable effect is a small regression. MimeTypes.guess_type already parses its argument as a URL and uses p.path (checked in the 3.11–3.14 stdlib sources and on the 3.12 runtime used here), so query strings and fragments were ignored before this change; tests/test_audio.py::test_signed_media_urls_reach_the_provider still passes with this line reverted.
The one input that behaves differently now is a URL whose path carries ; parameters: mimetypes.guess_type("https://cdn.example/a.wav;v=2") returned audio/x-wav, while urlsplit(...).path is a.wav;v=2 and yields None, so republic.image/audio/video(...) raises ValueError: Cannot guess the media type ... pass media_type explicitly for such URLs.
Keeping the extraction for explicitness is fine — just noting it adds no coverage, and dropping it preserves the behaviour documented above the loader.
|
/landing fix |
The standard library reports the legacy aliases `audio/x-wav` and `audio/x-aiff`
for `.wav` and `.aiff` files, and Google accepts only the canonical names they
stand for, so the documented `republic.audio("audio.wav")` example sent a MIME
type outside Gemini's supported audio set. Canonicalize both aliases when
writing `inlineData` and `fileData` MIME types; Chat Completions keeps receiving
the alias and maps it to its `wav` format label.
|
Repaired in dc9e8c3, now the head of this branch. The Gemini encoder canonicalizes the legacy aliases before writing Validation on dc9e8c3:
The separate non-blocking note on |
2 participants
Republic represents audio input with
Audioandaudio(), using the same MIME type and byte/URL source contract as Image and Video. Applications can pass these parts throughchat(),stream(), and attached conversation history.Gemini encodes inline audio and remote references. Chat Completions encodes inline WAV/MP3; OpenRouter extends the mapping for its documented audio formats. Unsupported format/source combinations raise
UnsupportedFeatureErrorbefore sending a request. Signed media URLs infer MIME types from their path, and the package includespy.typedfor downstream type checking.Validation:
ty checkpassed.