Skip to content

feat(cameras): ghost overlay of a saved reference frame to realign the rig - #139

Open
ravediamond wants to merge 1 commit into
huggingface:mainfrom
ravediamond:camera-reference-overlay
Open

ravediamond wants to merge 1 commit into
huggingface:mainfrom
ravediamond:camera-reference-overlay

Conversation

@ravediamond

@ravediamond ravediamond commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Closes #138

If you stash the arm and cameras between sessions, putting the framing back by eye is unreliable, and framing that drifts between episodes hurts policy training. The arm is in view, so one reference frame per camera captures both camera framing and arm pose.

In the recording dialog, each camera now has a Set reference button. It saves the frame currently shown in the preview. After that the reference is drawn as a translucent ghost over the live preview, with a slider for its opacity, so you can nudge the rig until the two line up. Clear removes it.

How it works:

  • Frontend only for the camera side. The frame is grabbed from the existing browser preview, so the backend never opens a camera and can't fight the browser or the recorder for the device.
  • The backend just stores the PNG: GET/PUT/DELETE /robots/{name}/camera-reference/{camera}, saved under ~/.cache/huggingface/lerobot/camera_references/<robot>/<camera>.png. Names go through the same check as robot names, and the body must be a PNG under 10 MB.
  • The ghost reuses the preview's rotation geometry, so it lines up with rotated cameras too.

Left out on purpose to keep this small: an alignment score and lighting match. They're in the standalone tool this came from (https://github.com/ravediamond/teleop-setup-align) and could follow if this is useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Camera reference overlay to realign the rig between sessions

1 participant