Skip to content

feat(datasets): move episodes out of a local dataset into a new one - #142

Open
ravediamond wants to merge 1 commit into
huggingface:mainfrom
ravediamond:feat/move-episodes
Open

ravediamond wants to merge 1 commit into
huggingface:mainfrom
ravediamond:feat/move-episodes

Conversation

@ravediamond

Copy link
Copy Markdown
Contributor

Follow-up to the note on #95: "removing episodes by moving them into a separate dataset" as its own small PR.

Problem

After a recording session some takes are bad (failed grasp, arm out of frame, forgotten reset), and there is no way to drop them from the browser. Today that means leaving LeLab for LeRobot's lerobot-edit-dataset.

Approach

A "Move episodes" button in the dataset browser, shown for local datasets with more than one episode. The dialog is prefilled with the episode being viewed and takes a list like 3, 7, 10-12 plus a name. Those episodes go to a new dataset local/<name>, and the episodes left in the source are renumbered. Nothing is deleted, so the moved takes can be merged back or kept apart.

  • POST /datasets/move-episodes wraps LeRobot's delete_episodes, run twice: once for the moved copy, once for the copy that stays. Both are written to a staging dir next to the output, and the source is swapped only once both exist, so a failure leaves the source untouched.
  • The route returns 409 while a recording is running. Empty or negative indices are a 422 at the request model; out-of-range indices, moving every episode, an invalid name or an existing output are a 400. ensure_episode_index runs before the dataset is opened, and the output path is contained under the cache root like merge's.
  • /dataset-frame and /dataset-video are now no-cache (videos revalidate against the FileResponse ETag). Both assumed a frame never changes for a given dataset/episode/index, which a move breaks.

If the dataset is also on the Hub, the local copy no longer matches it after a move.

Testing

  • pytest: new tests cover a real move on a LeRobot-written dataset, invalid requests, an existing output name, a failure mid-move that leaves the source untouched, the 409 while recording, and request validation.
  • vitest (dialog + episode-list parser), eslint, npm run build, pre-commit run --all-files.
  • Moved episodes through the HTTP route on a small recorded video dataset and decoded frames from both results: each kept its own footage, no staging dir left behind.
  • Drove the dialog end to end in headless Chromium.

Tested on Linux (Jetson Orin, aarch64) only. Not tested on macOS, Windows, or a real SO-101 dataset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant