Repository navigation
Is there a way to get response to image input? #1350
Description
Hi there,
I am currently working on Bidi Streaming using websockets. I am trying to have a image streaming feature too.
When I send images and audio(or text) together I got response. But I couldn't find a way to get response to only image data. Is there a way to do that?
In main.py, I handle image data like this:
elif mime_type.startswith("image"):
# Send image data (video frames)
decoded_data = base64.b64decode(data)
live_request_queue.send_realtime(Blob(data=decoded_data, mime_type=mime_type))
print(f"[CLIENT TO AGENT]: {mime_type}: {len(decoded_data)} bytes")
on app.js I added:
const startCamButton = document.getElementById('startCamButton')
startCamButton.addEventListener('click', async () => {
try {
startCamButton.disabled = true
startAudioButton.disabled = true
startAudio()
is_audio = true
connectWebsocket()
// Start video capture at fps FPS
await startVideoCapture(videoFrameHandler, fps)
} catch (error) {
console.error('Failed to start camera:', error)
startCamButton.disabled = false
alert(
'Failed to access camera. Please make sure you have granted camera permissions.'
)
}
})
// Video frame handler
function videoFrameHandler(frameData) {
// Send the frame data as base64
sendMessage({
mime_type: 'image/jpeg',
data: arrayBufferToBase64(frameData),
})
console.log('[CLIENT TO AGENT] sent video frame: bytes')
}
To app.js.
With these additions, I don't get response to only image data. For example when I say tell me the number of fingers you see continuously, I only got response when I talk. I'd like to send webcam frames and get responses like "You are showing three fingers" without having to speak or type anything.
Activity
hangfei commentedon Jul 2, 2025
You should use streaming tool: https://google.github.io/adk-docs/streaming/streaming-tools/