Skip to content

Commit b587792

Browse files
committed
so101 doc update
1 parent 80de585 commit b587792

7 files changed

Lines changed: 778 additions & 649 deletions

File tree

‎eval/lerobot_so101/README.md‎

Lines changed: 6 additions & 249 deletions
Original file line numberDiff line numberDiff line change
@@ -1,255 +1,12 @@
1-
<p align="center">
2-
<a href="README_zh.md">简体中文</a> | <strong>English</strong>
3-
</p>
1+
**English** | [简体中文](README_zh.md)
42

53
# LeRobot SO101
64

7-
This directory provides a real-robot synchronous control example for the
8-
LeRobot SO101 platform.
5+
Real-robot synchronous control for the LeRobot SO101 platform.
96

10-
In addition to the LeRobot SO101 follower arm, you need a camera. The default
11-
configuration uses an **Intel RealSense D435** (`CAMERA_TYPE=realsense`). On
12-
macOS you can alternatively use an iPhone as a wireless camera
13-
(`CAMERA_TYPE=iphone`, same Apple ID, confirm the OpenCV index).
14-
15-
## Usage
16-
17-
### step0: Environment
18-
19-
The conda environment name is `lerobot-demo`.
20-
21-
#### Clone and initialize submodules
22-
23-
```bash
24-
git clone https://github.com/Robot-cpp/robot.cpp
25-
cd robot.cpp
26-
git submodule update --init --recursive
27-
```
28-
29-
#### Create the Python environment
30-
31-
From the **repository root**.
32-
33-
**Option A: one-shot create**
34-
35-
```bash
36-
conda env create -f eval/lerobot_so101/environment.yaml
37-
conda activate lerobot-demo
38-
```
39-
40-
`environment.yaml` installs:
41-
42-
- local `third_party/lerobot[feetech,intelrealsense]` (SO101 serial ports, follower control, RealSense)
43-
- local `eval/lerobot_so101/lerobot_camera_opencv_crop` (`opencv_crop` / `realsense_crop` camera plugin)
44-
45-
**Option B: manual step-by-step install**
46-
47-
Use when `conda env create` hangs, pip fails, or you want to see install progress.
48-
49-
```bash
50-
conda create -n lerobot-demo python=3.12 pip -y
51-
conda activate lerobot-demo
52-
53-
pip install -e "third_party/lerobot[feetech,intelrealsense]"
54-
pip install -e eval/lerobot_so101/lerobot_camera_opencv_crop
55-
```
56-
57-
On **macOS**, install the RealSense Python package separately (`intelrealsense`
58-
from `environment.yaml` is unreliable on macOS):
59-
60-
```bash
61-
pip install -r eval/lerobot_so101/requirements-macos-realsense.txt
62-
bash eval/lerobot_so101/script/shell/fix_macos_pyrealsense_lib.sh
63-
```
64-
65-
Verify:
66-
67-
```bash
68-
python -c "import lerobot; from lerobot_camera_opencv_crop import RealSenseCameraCrop; print('ok')"
69-
# macOS also:
70-
python -c "import pyrealsense2 as rs; print('pyrealsense2 ok')"
71-
```
72-
73-
Notes:
74-
75-
- Initialize the `third_party/lerobot` submodule first.
76-
- Run the commands above from the repository root; pip `-e` paths are relative
77-
to the root.
78-
79-
### step1: Configure SO101
80-
81-
All real-robot scripts load environment defaults from:
82-
83-
| Platform | Config file |
7+
| Document | Description |
848
| --- | --- |
85-
| Linux / macOS | `eval/lerobot_so101/script/shell/so101_env.sh` |
86-
| Windows | `eval\lerobot_so101\script\bat\so101_env.bat` |
87-
88-
Edit serial ports, camera settings, and inference parameters for your machine:
89-
90-
| Variable | Description |
91-
| --- | --- |
92-
| `ROBOT_PORT` | SO101 follower serial port |
93-
| `TELEOP_PORT` | SO101 leader serial port; used for teleop, dataset recording, and leader calibration |
94-
| `CAMERA_TYPE` | `realsense` (default) or `iphone` |
95-
| `REALSENSE_SERIAL` | RealSense serial number, **required** when `CAMERA_TYPE=realsense` |
96-
| `CAMERA_KEY` | Camera field name in LeRobot observations; default `camera1` |
97-
| `MODEL_IMAGE_NAME` | Image key sent to model-server; must match GGUF checkpoint metadata (default `observation.images.camera1`) |
98-
| `CAMERA_INDEX` | OpenCV camera index or device path (only for `CAMERA_TYPE=iphone`) |
99-
| `CAMERA_WIDTH` / `CAMERA_HEIGHT` | Raw camera capture resolution |
100-
| `CAMERA_RESIZE_WIDTH` / `CAMERA_RESIZE_HEIGHT` | Resize size before model input; default 224×224 |
101-
| `ROBOT_PLATFORM` | Platform selector; default `lerobot_so101` |
102-
| `SERVER` | robot.cpp model-server address; default `127.0.0.1:5555` |
103-
| `TASK` | Natural-language task prompt sent to the model |
104-
| `FPS` | Synchronous control loop rate |
105-
106-
Find local serial ports:
107-
108-
```bash
109-
lerobot-find-port
110-
```
111-
112-
Find the RealSense serial number:
113-
114-
```bash
115-
python -m lerobot.scripts.lerobot_find_cameras realsense
116-
# macOS fallback: sudo rs-enumerate-devices -s
117-
```
118-
119-
To override settings temporarily without editing the file, export them in the
120-
current shell:
121-
122-
```bash
123-
export ROBOT_PORT=/dev/tty.usbmodemXXXX
124-
export TELEOP_PORT=/dev/tty.usbmodemYYYY
125-
export CAMERA_TYPE=realsense
126-
export REALSENSE_SERIAL=141722072266
127-
unset ROBOT_CAMERAS # clear cached camera JSON after changing camera settings
128-
```
129-
130-
Notes:
131-
132-
- `CAMERA_KEY` is the key in local LeRobot observations.
133-
- `MODEL_IMAGE_NAME` is the image name sent to model-server and must match the
134-
image key in the GGUF checkpoint (e.g. `observation.images.front` for some models).
135-
- The current SO101 client uses a single camera by default. For dual cameras,
136-
extend both the platform camera config and `RobotPolicy.build_observation`.
137-
138-
If the follower or leader has not been calibrated yet:
139-
140-
```bash
141-
cd eval/lerobot_so101
142-
./script/shell/calibrate_follower.sh
143-
./script/shell/calibrate_leader.sh
144-
```
145-
146-
After calibration, teleoperation is a good sanity check:
147-
148-
```bash
149-
cd eval/lerobot_so101
150-
./script/shell/teleoperate.sh
151-
```
152-
153-
### step2: Start model-server
154-
155-
The SO101 client does not load the model directly; it connects to a running
156-
robot.cpp server. Start the server from the repository root, for example:
157-
158-
```bash
159-
bash robot_server/shell/launch_robot_server_mac_cpu.sh
160-
```
161-
162-
For Windows or CUDA, use the matching script under `robot_server/shell/`.
163-
164-
### step3: Run the sync loop
165-
166-
After model-server is up, run in another terminal:
167-
168-
```bash
169-
bash eval/lerobot_so101/script/shell/run_robot_client.sh
170-
```
171-
172-
Windows:
173-
174-
```bat
175-
eval\lerobot_so101\script\bat\run_robot_client.bat
176-
```
177-
178-
This script sources `so101_env` and runs `run_sync.py`.
179-
180-
Runtime keys:
181-
182-
- `R`: clear the action queue and move the arm back to the home pose recorded at startup
183-
- `Q`: exit the sync control loop
184-
185-
### Camera test
186-
187-
For camera connectivity, RealSense drivers, and serial-number setup, see
188-
[camera_setup.md](camera_setup.md). Minimal smoke test:
189-
190-
**Linux / macOS** (macOS RealSense must run in a **root shell** via `sudo -s`,
191-
not `sudo bash`):
192-
193-
```bash
194-
cd eval/lerobot_so101
195-
unset ROBOT_CAMERAS
196-
export CAMERA_TYPE=realsense
197-
export REALSENSE_SERIAL=your_serial
198-
./test/run_camera_test.sh --preview
199-
```
200-
201-
**Windows (PowerShell)**:
202-
203-
```powershell
204-
cd eval\lerobot_so101
205-
$env:CAMERA_TYPE = "realsense"
206-
$env:REALSENSE_SERIAL = "your_serial"
207-
.\test\run_camera_test.bat
208-
```
209-
210-
This checks image shape, dtype, stride, and whether the constructed predict
211-
observation can be encoded by `model_client`.
212-
213-
## Current Layout
214-
215-
Main files:
216-
217-
```text
218-
eval/lerobot_so101/
219-
├── environment.yaml # conda env definition (lerobot-demo)
220-
├── camera_setup.md # RealSense / camera test guide
221-
├── requirements-macos-realsense.txt # macOS pyrealsense2 deps
222-
├── run_sync.py # real-robot entry: ModelClient + RobotPolicy + Platform + SyncControlLoop
223-
├── so101_client.py # SO101Platform: connect / get_observation / send_action / reset_home
224-
├── script/shell/so101_env.sh # serial ports, camera, server, task, fps (Linux/macOS)
225-
├── script/bat/so101_env.bat # Windows env (aligned with so101_env.sh)
226-
├── script/shell/build_robot_cameras.py # build ROBOT_CAMERAS JSON from env vars
227-
├── script/shell/check_realsense_env.sh # RealSense preflight checks
228-
├── script/shell/setup_macos_realsense.sh # macOS RealSense one-shot setup
229-
├── script/shell/fix_macos_pyrealsense_lib.sh
230-
├── script/shell/find_realsense.sh # list RealSense serial numbers
231-
├── script/shell/run_robot_client.sh # one-command sync loop launcher
232-
├── script/bat/run_robot_client.bat
233-
├── script/shell/calibrate_*.sh # LeRobot calibration scripts
234-
├── script/bat/calibrate_*.bat
235-
├── script/shell/teleoperate.sh # LeRobot teleop script
236-
├── script/bat/teleoperate.bat
237-
├── script/shell/record_dataset.sh # LeRobot dataset recording script
238-
├── script/bat/record_dataset.bat
239-
├── test/run_camera_test.sh # camera smoke test (Linux/macOS)
240-
├── test/run_camera_test.bat # camera smoke test (Windows)
241-
├── test/test_camera.py # camera and observation encoding checks
242-
├── utils/robot.py # camera JSON config and home pose helpers
243-
├── utils/stdin.py # non-blocking keyboard input
244-
└── lerobot_camera_opencv_crop/ # opencv_crop / realsense_crop camera plugin
245-
```
246-
247-
Data flow:
9+
| [setup_robot_en.md](setup_robot_en.md) | Environment, serial ports, calibration, model-server, sync loop |
10+
| [setup_camera_en.md](setup_camera_en.md) | Camera options, RealSense setup, connectivity test |
24811

249-
```text
250-
SO101Platform.get_observation()
251-
-> RobotPolicy.build_observation()
252-
-> ModelClient.predict()
253-
-> BasePolicy.select_action()
254-
-> SO101Platform.send_action()
255-
```
12+
Start with the [robot deployment guide](setup_robot_en.md).

0 commit comments

Comments
 (0)