Skip to content

fix(cameras): recover stalled browser streams - #147

Open
sysradium wants to merge 1 commit into
huggingface:mainfrom
sysradium:camerate-feed-fix
Open

sysradium wants to merge 1 commit into
huggingface:mainfrom
sysradium:camerate-feed-fix

Conversation

@sysradium

Copy link
Copy Markdown

On macOS in Firefox, the Innomaker U20CAM preview stayed grey even though macOS showed the camera as in use. It would open at 640脳480 but send no frames until another app (e.g. FaceTime) started it at a larger size. OpenCV gets frames at 640脳480, so recording was never affected; only the browser previews were.

Tested in Firefox with nothing else using the camera:

Requested size Frames in 3 s
1920脳1080 80
1280脳720 80
640脳480 0

So with this pr:

  • useCameraStream takes the camera's saved capture size and requests it first. If no frame arrives within 2.5 s, it reopens at 1280脳720, then 1920脳1080. All sizes are ideal constraints, so a camera that can't meet them still opens. Cameras that stream at their saved size never reach the fallback.

  • BrowserCameraPreview sizes its box from the saved capture size instead of the stream's actual size. cameraPreviewGeometry now uses object-fit: cover, so a stream at a different aspect ratio (e.g. a 16:9 fallback for a 4:3 recording) is center-cropped and the preview shows the framing that will be recorded.

  • Removed the metadata state (setMetadata on loadedmetadata) from BrowserCameraPreview. It stored the stream's actual videoWidth/videoHeight and sized the box from them, which is what made a 16:9 fallback stream show a wider view than the 4:3 recording. Now that the box always takes the saved size's shape, nothing reads the stream's size. Its deviceId check, which stopped a reassigned preview from reusing the previous camera's size, is no longer needed either. One consequence: a preview no longer shows when the browser picked a different size, because the image is cropped to the recorded framing.

  • Recording still captures at the saved size; LeRobot raises an error if it can't set it.

  • The recording page's live view (/camera-feed) already showed the recorded frames and is untouched.

  • Teleoperation doesn't open cameras on the backend.

Notes

  • For the Innomaker, the 640脳480 mode is a center crop of the sensor. Template matching OpenCV captures put it at the middle 1440脳1080 of the 1920脳1080 frame (x=236; exact center is 240). So the cropped fallback preview matches the recording for this camera. A camera that scales the full sensor down instead of cropping could differ slightly at the edges.
  • A camera that stalls takes about 2.5 s longer before its preview appears.

Request the recorded capture size first, then retry at fallback resolutions when the stream delivers no frames. Keep previews framed to the recorded aspect ratio and crop mismatched streams.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant