Skip to content

Keep recordings on Ctrl+C, drop dataset repair, make Stop immediate - #134

Merged
nicolas-rabault merged 2 commits into
mainfrom
fix/finalize-on-shutdown
Oct 8, 2026
Merged

nicolas-rabault merged 2 commits into
mainfrom
fix/finalize-on-shutdown

Conversation

@nicolas-rabault

Copy link
Copy Markdown
Member

Ctrl+C on the lelab server during a recording destroyed the whole session, not just the episode in progress. This PR saves it, replaces the dataset repair added in #84 with a clear error, and makes Stop end the session right away.

Why the session was lost

The recording worker is a daemon=True thread. On Ctrl+C, Python killed it without running its finally, so LeRobotDataset.finalize() never ran. LeRobot only finalizes two things there:

  • the data parquet: one ParquetWriter stays open until 100 MB or finalize(), and a parquet file without its footer cannot be read;
  • meta/episodes/: kept in memory until finalize().

A SO-101 session writes a few MB per hour, so its data file is open for the whole session. After a kill, nothing in it is readable. LeRobot then takes the dataset for one not downloaded yet, asks the Hub, and gets a 404.

Changes

  • Finalize on shutdown. stop_recording_and_wait() takes the same path as the Stop button and waits up to 60 s for the session to finalize. The FastAPI shutdown hook calls it, and so does the Windows Ctrl+C handler, which calls os._exit and skips that hook. lelab --dev gets the same 2 s --timeout-graceful-shutdown as prod, because an open /camera-feed stream otherwise keeps uvicorn from ever reaching the hook.
  • Drop dataset_repair.py. It rebuilt meta/episodes/ from the data files, but after a real kill those files have no footer either, so it recovered nothing for sessions LeLab records. I checked this with an actual kill, not just by reading the code. ensure_episode_index() in episode_media.py replaces it on dataset-info, upload, merge and "Record more". It returns "has no readable episodes, most likely because its recording was interrupted. Delete it and record it again." instead of the Hub 404, and it still refuses a repo_id that escapes the cache. fix(record): preserve video chunk identity in repaired statistics聽#122's fix goes away with the module.
  • Faster Stop. Pressing Stop during an episode went through the re-record branch and then sat out a full reset phase, with the arm still following the leader and the UI showing "resetting". It now ends right after discarding the take. LeRobot's own ESC also skips the reset. A Stop sent during device setup no longer waits out the first reset either. The take in progress is still discarded, as the confirm dialog says ("Saved episodes are kept").

Validation

A harness runs the real server with fake arms and a fake camera (streaming video encoding):

Scenario Before After
Ctrl+C during episode 3 (2 saved) data parquet unreadable, no index, Hub 404 2 episodes, dataset opens
Stop during episode 3 (reset_time_s=8) 9.3 s 1.2 s, 2 episodes, video frames = data rows
Stop / Ctrl+C during device setup ~10 s (full first reset) 1.7 s / 2.6 s
Next then Stop, Re-record then Stop 2 and 1 episodes kept
kill -9, then /dataset-info and /upload-dataset Hub 404 the message above
../outside/ds on /dataset-info refused

An independent review also probed Stop during save_episode(), Stop during the first reset, batch encoding, and a double Ctrl+C. All 290 tests and pre-commit pass. The timings above come from an idle machine. After rebasing on 08b1676, every scenario passed again on a heavily loaded one (load average 30 to 90), only slower.

Not tested: Windows, and real hardware. The only physical difference from Stop is that the follower loses torque right away instead of after the reset time. That is already what happens after the last episode.

Known limits, unchanged by this PR

  • A kill -9 or power cut still loses the session. Only LeRobot could avoid that, by writing its files so they survive a crash.
  • Dev mode's launcher still kills uvicorn 5 s after Ctrl+C.
  • A resumed dataset killed mid-session keeps its old, readable index, so the check lets it through and LeRobot raises a raw parquet error. The removed repair had the same gap.

The recording worker is a daemon thread, so Ctrl+C on the server killed it
before its teardown ran. finalize() never wrote the parquet footers or the
episode index, and the whole session became unreadable: LeRobot took it
for a dataset not downloaded yet and asked the Hub, which 404s. The
FastAPI shutdown hook, and the Windows Ctrl+C handler that bypasses it,
now stop the session and wait for it to finalize. Dev mode gets the same
2 s graceful-shutdown timeout as prod, so an open /camera-feed stream no
longer holds that hook off.

dataset_repair rebuilt the episode index from the data files. After a
real hard kill those files have no footer either, because LeRobot keeps
one ParquetWriter open until 100 MB or finalize, so it recovered nothing
for the sessions LeLab records. ensure_episode_index replaces it: it turns
the Hub 404 into a message saying the recording was interrupted and has
to be recorded again, and still refuses repo_ids that escape the cache.

Stop pressed mid-episode went through the re-record branch and sat out a
full reset phase before ending. It now ends right after discarding the
take, like LeRobot's own ESC, and a Stop sent during device setup no
longer waits out the first reset either. Saved episodes and the
discarded take behave as before.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 11:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃煛 Changes recommended

Missing metadata bypasses cache containment validation, allowing outside-cache paths to reach LeRobot.

1 open finding
What changed in this PR

This PR preserves saved LeLab recording episodes during graceful shutdown and shortens Stop handling.

Changes:

  • Wait for recording finalization during shutdown.
  • Replace dataset repair with explicit unreadable-index errors.
  • Skip unnecessary reset phases after Stop and update regression tests.
File Description
tests/鈥媡est_server.py Tests shutdown finalization.
tests/鈥媡est_record.py Tests shutdown waiting and resume validation.
tests/鈥媡est_episode_media.py Tests episode-index validation.
tests/鈥媡est_datasets.py Replaces repair tests with rejection checks.
tests/鈥媡est_dataset_repair.py Removes repair tests.
lelab/鈥媠erver.py Waits for recordings during shutdown.
lelab/鈥媠cripts/鈥媗elab.py Updates Windows and development shutdown handling.
lelab/鈥媟ecord.py Adds finalization waiting and faster Stop handling.
lelab/鈥媏pisode_media.py Adds unreadable-index validation.
lelab/鈥媎atasets.py Validates merge sources without repair.
lelab/鈥媎ataset_repair.py Removes automatic dataset repair.
CLAUDE.md Documents shutdown and validation behavior.

馃 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread lelab/episode_media.py Outdated
ensure_episode_index only checked cache containment once it had found a
meta/info.json, so an id escaping the cache with nothing at its target
(../outside/absent, an absolute path, the cache root, a symlink pointing
out) went straight to LeRobot, and a "Record more" resume built its root
outside the cache. The removed repair rejected those unconditionally.
Containment is now checked first, through the same helper
resolve_dataset_dir uses.
@nicolas-rabault
nicolas-rabault merged commit fe661bb into main Oct 8, 2026
6 checks passed
@nicolas-rabault
nicolas-rabault deleted the fix/finalize-on-shutdown branch October 8, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants