Repository navigation
Keep recordings on Ctrl+C, drop dataset repair, make Stop immediate - #134
Merged
Merged
Conversation
The recording worker is a daemon thread, so Ctrl+C on the server killed it before its teardown ran. finalize() never wrote the parquet footers or the episode index, and the whole session became unreadable: LeRobot took it for a dataset not downloaded yet and asked the Hub, which 404s. The FastAPI shutdown hook, and the Windows Ctrl+C handler that bypasses it, now stop the session and wait for it to finalize. Dev mode gets the same 2 s graceful-shutdown timeout as prod, so an open /camera-feed stream no longer holds that hook off. dataset_repair rebuilt the episode index from the data files. After a real hard kill those files have no footer either, because LeRobot keeps one ParquetWriter open until 100 MB or finalize, so it recovered nothing for the sessions LeLab records. ensure_episode_index replaces it: it turns the Hub 404 into a message saying the recording was interrupted and has to be recorded again, and still refuses repo_ids that escape the cache. Stop pressed mid-episode went through the re-record branch and sat out a full reset phase before ending. It now ends right after discarding the take, like LeRobot's own ESC, and a Stop sent during device setup no longer waits out the first reset either. Saved episodes and the discarded take behave as before.
There was a problem hiding this comment.
馃煛 Changes recommended
Missing metadata bypasses cache containment validation, allowing outside-cache paths to reach LeRobot.
1 open finding
What changed in this PR
This PR preserves saved LeLab recording episodes during graceful shutdown and shortens Stop handling.
Changes:
- Wait for recording finalization during shutdown.
- Replace dataset repair with explicit unreadable-index errors.
- Skip unnecessary reset phases after Stop and update regression tests.
| File | Description |
|---|---|
| tests/鈥媡est_server.py | Tests shutdown finalization. |
| tests/鈥媡est_record.py | Tests shutdown waiting and resume validation. |
| tests/鈥媡est_episode_media.py | Tests episode-index validation. |
| tests/鈥媡est_datasets.py | Replaces repair tests with rejection checks. |
| tests/鈥媡est_dataset_repair.py | Removes repair tests. |
| lelab/鈥媠erver.py | Waits for recordings during shutdown. |
| lelab/鈥媠cripts/鈥媗elab.py | Updates Windows and development shutdown handling. |
| lelab/鈥媟ecord.py | Adds finalization waiting and faster Stop handling. |
| lelab/鈥媏pisode_media.py | Adds unreadable-index validation. |
| lelab/鈥媎atasets.py | Validates merge sources without repair. |
| lelab/鈥媎ataset_repair.py | Removes automatic dataset repair. |
| CLAUDE.md | Documents shutdown and validation behavior. |
馃 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
ensure_episode_index only checked cache containment once it had found a meta/info.json, so an id escaping the cache with nothing at its target (../outside/absent, an absolute path, the cache root, a symlink pointing out) went straight to LeRobot, and a "Record more" resume built its root outside the cache. The removed repair rejected those unconditionally. Containment is now checked first, through the same helper resolve_dataset_dir uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Ctrl+C on the
lelabserver during a recording destroyed the whole session, not just the episode in progress. This PR saves it, replaces the dataset repair added in #84 with a clear error, and makes Stop end the session right away.Why the session was lost
The recording worker is a
daemon=Truethread. On Ctrl+C, Python killed it without running itsfinally, soLeRobotDataset.finalize()never ran. LeRobot only finalizes two things there:ParquetWriterstays open until 100 MB orfinalize(), and a parquet file without its footer cannot be read;meta/episodes/: kept in memory untilfinalize().A SO-101 session writes a few MB per hour, so its data file is open for the whole session. After a kill, nothing in it is readable. LeRobot then takes the dataset for one not downloaded yet, asks the Hub, and gets a 404.
Changes
stop_recording_and_wait()takes the same path as the Stop button and waits up to 60 s for the session to finalize. The FastAPI shutdown hook calls it, and so does the Windows Ctrl+C handler, which callsos._exitand skips that hook.lelab --devgets the same 2 s--timeout-graceful-shutdownas prod, because an open/camera-feedstream otherwise keeps uvicorn from ever reaching the hook.dataset_repair.py. It rebuiltmeta/episodes/from the data files, but after a real kill those files have no footer either, so it recovered nothing for sessions LeLab records. I checked this with an actual kill, not just by reading the code.ensure_episode_index()inepisode_media.pyreplaces it on dataset-info, upload, merge and "Record more". It returns "has no readable episodes, most likely because its recording was interrupted. Delete it and record it again." instead of the Hub 404, and it still refuses arepo_idthat escapes the cache. fix(record): preserve video chunk identity in repaired statistics聽#122's fix goes away with the module.Validation
A harness runs the real server with fake arms and a fake camera (streaming video encoding):
reset_time_s=8)kill -9, then/dataset-infoand/upload-dataset../outside/dson/dataset-infoAn independent review also probed Stop during
save_episode(), Stop during the first reset, batch encoding, and a double Ctrl+C. All 290 tests and pre-commit pass. The timings above come from an idle machine. After rebasing on 08b1676, every scenario passed again on a heavily loaded one (load average 30 to 90), only slower.Not tested: Windows, and real hardware. The only physical difference from Stop is that the follower loses torque right away instead of after the reset time. That is already what happens after the last episode.
Known limits, unchanged by this PR
kill -9or power cut still loses the session. Only LeRobot could avoid that, by writing its files so they survive a crash.