|
1 | | -<p align="center"> |
2 | | - <a href="README_zh.md">简体中文</a> | <strong>English</strong> |
3 | | -</p> |
| 1 | +**English** | [简体中文](README_zh.md) |
4 | 2 |
|
5 | 3 | # LeRobot SO101 |
6 | 4 |
|
7 | | -This directory provides a real-robot synchronous control example for the |
8 | | -LeRobot SO101 platform. |
| 5 | +Real-robot synchronous control for the LeRobot SO101 platform. |
9 | 6 |
|
10 | | -In addition to the LeRobot SO101 follower arm, you need a camera. The default |
11 | | -configuration uses an **Intel RealSense D435** (`CAMERA_TYPE=realsense`). On |
12 | | -macOS you can alternatively use an iPhone as a wireless camera |
13 | | -(`CAMERA_TYPE=iphone`, same Apple ID, confirm the OpenCV index). |
14 | | - |
15 | | -## Usage |
16 | | - |
17 | | -### step0: Environment |
18 | | - |
19 | | -The conda environment name is `lerobot-demo`. |
20 | | - |
21 | | -#### Clone and initialize submodules |
22 | | - |
23 | | -```bash |
24 | | -git clone https://github.com/Robot-cpp/robot.cpp |
25 | | -cd robot.cpp |
26 | | -git submodule update --init --recursive |
27 | | -``` |
28 | | - |
29 | | -#### Create the Python environment |
30 | | - |
31 | | -From the **repository root**. |
32 | | - |
33 | | -**Option A: one-shot create** |
34 | | - |
35 | | -```bash |
36 | | -conda env create -f eval/lerobot_so101/environment.yaml |
37 | | -conda activate lerobot-demo |
38 | | -``` |
39 | | - |
40 | | -`environment.yaml` installs: |
41 | | - |
42 | | -- local `third_party/lerobot[feetech,intelrealsense]` (SO101 serial ports, follower control, RealSense) |
43 | | -- local `eval/lerobot_so101/lerobot_camera_opencv_crop` (`opencv_crop` / `realsense_crop` camera plugin) |
44 | | - |
45 | | -**Option B: manual step-by-step install** |
46 | | - |
47 | | -Use when `conda env create` hangs, pip fails, or you want to see install progress. |
48 | | - |
49 | | -```bash |
50 | | -conda create -n lerobot-demo python=3.12 pip -y |
51 | | -conda activate lerobot-demo |
52 | | - |
53 | | -pip install -e "third_party/lerobot[feetech,intelrealsense]" |
54 | | -pip install -e eval/lerobot_so101/lerobot_camera_opencv_crop |
55 | | -``` |
56 | | - |
57 | | -On **macOS**, install the RealSense Python package separately (`intelrealsense` |
58 | | -from `environment.yaml` is unreliable on macOS): |
59 | | - |
60 | | -```bash |
61 | | -pip install -r eval/lerobot_so101/requirements-macos-realsense.txt |
62 | | -bash eval/lerobot_so101/script/shell/fix_macos_pyrealsense_lib.sh |
63 | | -``` |
64 | | - |
65 | | -Verify: |
66 | | - |
67 | | -```bash |
68 | | -python -c "import lerobot; from lerobot_camera_opencv_crop import RealSenseCameraCrop; print('ok')" |
69 | | -# macOS also: |
70 | | -python -c "import pyrealsense2 as rs; print('pyrealsense2 ok')" |
71 | | -``` |
72 | | - |
73 | | -Notes: |
74 | | - |
75 | | -- Initialize the `third_party/lerobot` submodule first. |
76 | | -- Run the commands above from the repository root; pip `-e` paths are relative |
77 | | - to the root. |
78 | | - |
79 | | -### step1: Configure SO101 |
80 | | - |
81 | | -All real-robot scripts load environment defaults from: |
82 | | - |
83 | | -| Platform | Config file | |
| 7 | +| Document | Description | |
84 | 8 | | --- | --- | |
85 | | -| Linux / macOS | `eval/lerobot_so101/script/shell/so101_env.sh` | |
86 | | -| Windows | `eval\lerobot_so101\script\bat\so101_env.bat` | |
87 | | - |
88 | | -Edit serial ports, camera settings, and inference parameters for your machine: |
89 | | - |
90 | | -| Variable | Description | |
91 | | -| --- | --- | |
92 | | -| `ROBOT_PORT` | SO101 follower serial port | |
93 | | -| `TELEOP_PORT` | SO101 leader serial port; used for teleop, dataset recording, and leader calibration | |
94 | | -| `CAMERA_TYPE` | `realsense` (default) or `iphone` | |
95 | | -| `REALSENSE_SERIAL` | RealSense serial number, **required** when `CAMERA_TYPE=realsense` | |
96 | | -| `CAMERA_KEY` | Camera field name in LeRobot observations; default `camera1` | |
97 | | -| `MODEL_IMAGE_NAME` | Image key sent to model-server; must match GGUF checkpoint metadata (default `observation.images.camera1`) | |
98 | | -| `CAMERA_INDEX` | OpenCV camera index or device path (only for `CAMERA_TYPE=iphone`) | |
99 | | -| `CAMERA_WIDTH` / `CAMERA_HEIGHT` | Raw camera capture resolution | |
100 | | -| `CAMERA_RESIZE_WIDTH` / `CAMERA_RESIZE_HEIGHT` | Resize size before model input; default 224×224 | |
101 | | -| `ROBOT_PLATFORM` | Platform selector; default `lerobot_so101` | |
102 | | -| `SERVER` | robot.cpp model-server address; default `127.0.0.1:5555` | |
103 | | -| `TASK` | Natural-language task prompt sent to the model | |
104 | | -| `FPS` | Synchronous control loop rate | |
105 | | - |
106 | | -Find local serial ports: |
107 | | - |
108 | | -```bash |
109 | | -lerobot-find-port |
110 | | -``` |
111 | | - |
112 | | -Find the RealSense serial number: |
113 | | - |
114 | | -```bash |
115 | | -python -m lerobot.scripts.lerobot_find_cameras realsense |
116 | | -# macOS fallback: sudo rs-enumerate-devices -s |
117 | | -``` |
118 | | - |
119 | | -To override settings temporarily without editing the file, export them in the |
120 | | -current shell: |
121 | | - |
122 | | -```bash |
123 | | -export ROBOT_PORT=/dev/tty.usbmodemXXXX |
124 | | -export TELEOP_PORT=/dev/tty.usbmodemYYYY |
125 | | -export CAMERA_TYPE=realsense |
126 | | -export REALSENSE_SERIAL=141722072266 |
127 | | -unset ROBOT_CAMERAS # clear cached camera JSON after changing camera settings |
128 | | -``` |
129 | | - |
130 | | -Notes: |
131 | | - |
132 | | -- `CAMERA_KEY` is the key in local LeRobot observations. |
133 | | -- `MODEL_IMAGE_NAME` is the image name sent to model-server and must match the |
134 | | - image key in the GGUF checkpoint (e.g. `observation.images.front` for some models). |
135 | | -- The current SO101 client uses a single camera by default. For dual cameras, |
136 | | - extend both the platform camera config and `RobotPolicy.build_observation`. |
137 | | - |
138 | | -If the follower or leader has not been calibrated yet: |
139 | | - |
140 | | -```bash |
141 | | -cd eval/lerobot_so101 |
142 | | -./script/shell/calibrate_follower.sh |
143 | | -./script/shell/calibrate_leader.sh |
144 | | -``` |
145 | | - |
146 | | -After calibration, teleoperation is a good sanity check: |
147 | | - |
148 | | -```bash |
149 | | -cd eval/lerobot_so101 |
150 | | -./script/shell/teleoperate.sh |
151 | | -``` |
152 | | - |
153 | | -### step2: Start model-server |
154 | | - |
155 | | -The SO101 client does not load the model directly; it connects to a running |
156 | | -robot.cpp server. Start the server from the repository root, for example: |
157 | | - |
158 | | -```bash |
159 | | -bash robot_server/shell/launch_robot_server_mac_cpu.sh |
160 | | -``` |
161 | | - |
162 | | -For Windows or CUDA, use the matching script under `robot_server/shell/`. |
163 | | - |
164 | | -### step3: Run the sync loop |
165 | | - |
166 | | -After model-server is up, run in another terminal: |
167 | | - |
168 | | -```bash |
169 | | -bash eval/lerobot_so101/script/shell/run_robot_client.sh |
170 | | -``` |
171 | | - |
172 | | -Windows: |
173 | | - |
174 | | -```bat |
175 | | -eval\lerobot_so101\script\bat\run_robot_client.bat |
176 | | -``` |
177 | | - |
178 | | -This script sources `so101_env` and runs `run_sync.py`. |
179 | | - |
180 | | -Runtime keys: |
181 | | - |
182 | | -- `R`: clear the action queue and move the arm back to the home pose recorded at startup |
183 | | -- `Q`: exit the sync control loop |
184 | | - |
185 | | -### Camera test |
186 | | - |
187 | | -For camera connectivity, RealSense drivers, and serial-number setup, see |
188 | | -[camera_setup.md](camera_setup.md). Minimal smoke test: |
189 | | - |
190 | | -**Linux / macOS** (macOS RealSense must run in a **root shell** via `sudo -s`, |
191 | | -not `sudo bash`): |
192 | | - |
193 | | -```bash |
194 | | -cd eval/lerobot_so101 |
195 | | -unset ROBOT_CAMERAS |
196 | | -export CAMERA_TYPE=realsense |
197 | | -export REALSENSE_SERIAL=your_serial |
198 | | -./test/run_camera_test.sh --preview |
199 | | -``` |
200 | | - |
201 | | -**Windows (PowerShell)**: |
202 | | - |
203 | | -```powershell |
204 | | -cd eval\lerobot_so101 |
205 | | -$env:CAMERA_TYPE = "realsense" |
206 | | -$env:REALSENSE_SERIAL = "your_serial" |
207 | | -.\test\run_camera_test.bat |
208 | | -``` |
209 | | - |
210 | | -This checks image shape, dtype, stride, and whether the constructed predict |
211 | | -observation can be encoded by `model_client`. |
212 | | - |
213 | | -## Current Layout |
214 | | - |
215 | | -Main files: |
216 | | - |
217 | | -```text |
218 | | -eval/lerobot_so101/ |
219 | | -├── environment.yaml # conda env definition (lerobot-demo) |
220 | | -├── camera_setup.md # RealSense / camera test guide |
221 | | -├── requirements-macos-realsense.txt # macOS pyrealsense2 deps |
222 | | -├── run_sync.py # real-robot entry: ModelClient + RobotPolicy + Platform + SyncControlLoop |
223 | | -├── so101_client.py # SO101Platform: connect / get_observation / send_action / reset_home |
224 | | -├── script/shell/so101_env.sh # serial ports, camera, server, task, fps (Linux/macOS) |
225 | | -├── script/bat/so101_env.bat # Windows env (aligned with so101_env.sh) |
226 | | -├── script/shell/build_robot_cameras.py # build ROBOT_CAMERAS JSON from env vars |
227 | | -├── script/shell/check_realsense_env.sh # RealSense preflight checks |
228 | | -├── script/shell/setup_macos_realsense.sh # macOS RealSense one-shot setup |
229 | | -├── script/shell/fix_macos_pyrealsense_lib.sh |
230 | | -├── script/shell/find_realsense.sh # list RealSense serial numbers |
231 | | -├── script/shell/run_robot_client.sh # one-command sync loop launcher |
232 | | -├── script/bat/run_robot_client.bat |
233 | | -├── script/shell/calibrate_*.sh # LeRobot calibration scripts |
234 | | -├── script/bat/calibrate_*.bat |
235 | | -├── script/shell/teleoperate.sh # LeRobot teleop script |
236 | | -├── script/bat/teleoperate.bat |
237 | | -├── script/shell/record_dataset.sh # LeRobot dataset recording script |
238 | | -├── script/bat/record_dataset.bat |
239 | | -├── test/run_camera_test.sh # camera smoke test (Linux/macOS) |
240 | | -├── test/run_camera_test.bat # camera smoke test (Windows) |
241 | | -├── test/test_camera.py # camera and observation encoding checks |
242 | | -├── utils/robot.py # camera JSON config and home pose helpers |
243 | | -├── utils/stdin.py # non-blocking keyboard input |
244 | | -└── lerobot_camera_opencv_crop/ # opencv_crop / realsense_crop camera plugin |
245 | | -``` |
246 | | - |
247 | | -Data flow: |
| 9 | +| [setup_robot_en.md](setup_robot_en.md) | Environment, serial ports, calibration, model-server, sync loop | |
| 10 | +| [setup_camera_en.md](setup_camera_en.md) | Camera options, RealSense setup, connectivity test | |
248 | 11 |
|
249 | | -```text |
250 | | -SO101Platform.get_observation() |
251 | | - -> RobotPolicy.build_observation() |
252 | | - -> ModelClient.predict() |
253 | | - -> BasePolicy.select_action() |
254 | | - -> SO101Platform.send_action() |
255 | | -``` |
| 12 | +Start with the [robot deployment guide](setup_robot_en.md). |
0 commit comments