Pounce — clutch-driven VR teleoperation for xLeRobot · xLeRobot 的离合式 VR 遥操作与 STS 舵机工具
一个 Windows 桌面程序（Python + tkinter）：用 Oculus Rift S 的 Touch 手柄通过 SteamVR 直接遥操作 xLeRobot 的双臂、四轮底盘和相机云台；同时内置 Feetech STS 总线舵机的 调试 / 编程 / 中位校准工具（可替代原厂 FD.exe）。深色界面，中英双语，状态栏里住着 像素猫 Cali。
A Windows desktop app (Python + tkinter) that teleoperates an xLeRobot's two SO-101 arms, four-wheel base and camera gimbal from Oculus Rift S Touch controllers through SteamVR, and doubles as a Feetech STS bus servo debugger, programmer and midpoint calibrator (a replacement for the factory FD.exe). Dark UI, bilingual, with Cali the pixel cat in the status bar.
your hand
→ headset cameras (position) + controller IMU (orientation)
→ SteamVR → this app reads poses at 60 Hz
→ drift gate → clutch delta → room→arm rotation → One Euro filter → IK
→ joint commands at 20 Hz over two USB-serial buses → STS3215 servos
- 离合式控制 Clutch-delta control — squeeze the grip and move your hand, the gripper follows; release and the arm freezes; re-grab and it continues from where it stopped, like lifting a mouse off the desk. The arm can never jump.
- 末端模式 End-effector (IK) mode — closed-form inverse kinematics for the SO-101 geometry: push forward and the gripper moves in a straight line. Joint mode stays available as a fallback.
- 测量而非假设 Measure, don't assume — guided wizards measure the operator's "forward" (Cal-Wizard), each motor's direction and zero pose (Servo-Cal) and the wheel layout (Wheel-ID) instead of hard-coding them.
- 为消费级追踪设计的滤波 Filtering for inside-out tracking — a drift gate absorbs tracker drift and teleports, a One Euro filter removes jitter without adding lag, soft deadzones never snap.
- 多层安全 Layered safety — e-stop from keyboard, controller and GUI; reach box, floor limit, workspace clamp and rate limits in the IK path; a servo watchdog for bus voltage, overload and temperature.
- 头显里看相机 Camera in the headset — a Logitech C920 stream drawn into the headset, with the camera gimbal following your head.
- 双臂完全独立 Both arms independent, tank drive from the controllers, hold-to-move gripper on the trigger, every keyboard key and controller input rebindable from the GUI.
- 可在主机上测试的核心 Host-testable core — protocol, kinematics, filters, binding models and the SteamVR data files are pure code covered by 283 unit tests that need no hardware.
The tool was developed on, and its shipped configuration is tuned for, one
specific build. Adapt teleop_config.json for yours.
| Part | What the code expects |
|---|---|
| Robot | xLeRobot: two SO-101-class arms, 4-wheel base, 2-axis camera gimbal, all Feetech STS3215 (12 V) |
| Servo buses | Two USB-serial adapters at 1,000,000 bps. Bus A ("left hand"): arm IDs 1–6 + wheels 7–10. Bus B ("right hand"): arm IDs 1–6 + gimbal pan 7 / tilt 8 |
| VR | Oculus Rift S (tethered) with Touch controllers, through the Oculus PC software and SteamVR |
| Camera (optional) | Logitech C920 on the same PC, shown in the headset through OpenGL |
| PC | Windows 10/11, Python 3.9+ (developed on 3.9.13) |
The servo pages (Debug, Program, OneKey, PWM) speak the plain Feetech SCS/STS bus protocol, so they work with any STS-servo build, not only xLeRobot.
git clone https://github.com/<your-account>/pounce.git
cd pounce
python -m pip install -r requirements.txt
python main.pyrun.batinstalls only the minimal dependencies (pyserial + openvr) and starts the app without a console window.build_exe.batinstalls everything, runs the tests and packs a standalonedist\Pounce.exewith PyInstaller (see Download below).- Dependencies:
pyserial(servo bus),openvr(SteamVR),opencv-python+glfw+PyOpenGL(only the camera-in-headset feature),pyinstaller(only to build the exe). The app starts without the optional ones.
Then follow docs/VR_QUICKSTART.md: power and plug, start Oculus → SteamVR → the app, open both serial ports in the top bar, arm the hands on the Control tab, connect and enable VR on the VR tab. Servo-level setup (finding IDs, midpoint calibration, angle limits) is in docs/MANUAL.md.
Beginners can skip the Python setup entirely. Pounce.exe is in this
repository (about 65 MB): open it on GitHub and click Download raw file,
or clone the repo. Put it in a folder of its own and double-click it.
On the first start it creates teleop_config.json next to itself; the session
log and dump files land in the same folder. Everything the app needs is inside
the exe: Python, the OpenVR client library, the camera stack and the fonts.
For VR you still need the Oculus PC software and SteamVR installed and running.
Two things Windows may do the first time. SmartScreen warns about unsigned
downloads: click "More info", then "Run anyway". Windows Defender may also
flag the file, which is a known false positive for unsigned PyInstaller
executables rather than anything in this code. The exe is produced from this
repository by build_exe.bat, so you can rebuild it yourself and compare.
VR (default vla profile, both controllers independent — left = arm A, right = arm B)
| Input | Action |
|---|---|
| Grip (hold) | Clutch: the arm follows your hand while held, freezes on release |
| Hand position / wrist pitch / wrist roll while clutched | EE mode: gripper position / gripper pitch / wrist roll (J5). Joint mode: wrist yaw → J1, pitch → J3, roll → J5 |
| Trigger (hold) | That side's wheel pair forward |
| A / X (hold) | That side's wheel pair backward |
| Thumbstick left / right | Gripper close / open |
| B / Y | E-stop — torque off on every servo of both buses (always, not rebindable) |
Every other input is rebindable from the VR tab ("Rebind"); Y/B stay the e-stop. The camera card on the VR tab shows the C920 in the headset and lets the gimbal follow your head.
Keyboard (Control tab): arm A Q/A W/S E/D R/F T/G Y/H, arm B
U/J I/K O/L P/; [/' ]/\, base arrows, camera Z/X C/V. Keyboard hybrid on
the VR tab: F / J = clutch A / B, C/V and N/M = grippers. Esc is
always the e-stop. Keys are editable on the Control tab (KEY MAP → Edit keys).
teleop_config.json beside main.py is created with defaults on the first
run and merged with any new keys on upgrade; edit it and click Reload on the
Control tab. The copy in this repository is the author's calibrated
configuration (servo IDs, wheel grouping, gains, kinematics signs and zero
pose, rest poses, operator frame). Use it as a reference, but run the wizards
on your own robot before trusting EE mode: Wheel-ID (Control tab), Servo-Cal
and Cal-Wizard (VR tab). See docs/EE_MODE.md.
Files the app writes beside main.py (or beside the exe), all ignored by git:
session_log.txt (the log pane), teleop_ee_log.csv (when "Log FK" is
ticked; replay with python tools/replay_fk.py teleop_ee_log.csv --plot),
crash_dump.txt / freeze_dump.txt (every thread's stack after a crash or a
GUI freeze), xlerobot_cali.vrmanifest (SteamVR app registration, rewritten
on every VR connect).
- Torque off before moving a joint by hand; un-tick torque immediately if a servo stalls against a hard stop.
- First VR run: wheels off the ground, clear space around the arms, a thumb near B/Y.
- E-stop =
Esc, B or Y on the controllers, or the GUI button: every servo on both buses loses torque. - EE mode must not run at full speed before Servo-Cal; the config flag
kinematics.calibratedgates a warning. - "The arms lost power" is usually the servos' own overload protection after the 12 V rail dropped out (the Hands line on the VR tab shows each bus voltage). Check the supply before the software.
Pounce.exe prebuilt Windows executable, no Python needed
main.py entry point (python main.py)
fdstudio/ the application package
protocol.py Feetech SCS/STS packet framing, pure functions
memory_map.py STS3215 register table (EPROM / SRAM, bilingual)
bus.py serial transactions on one worker thread
kinematics.py FK, closed-form IK, One Euro filter, limiters
vr_input.py OpenVR poses, action-manifest input, DriftGate
vrcam.py C920 → headset feed, head-following gimbal math
ctrlmap.py / keymap.py controller / keyboard binding models
watchdog.py freeze detector and crash dumps
gui.py tkinter GUI: tabs, teleop loop, wizards
tests/ unit tests (python -m unittest discover -s tests)
tools/replay_fk.py replay a logged session's FK offline
tools/vr_probe.py standalone controller input probe (run with the GUI closed)
docs/ manuals, mathematics, design plan, development log
vr_actions.json SteamVR action manifest
vr_bindings_oculus_touch.json default Touch bindings
teleop_config.json reference configuration
assets/ icons (window, exe) and button images
fonts/ Montserrat (SIL Open Font License, fonts/OFL.txt)
run.bat / build_exe.bat run from source / build a standalone exe
python -m unittest discover -s tests -v283 tests. They need the pip dependencies but no robot, no headset and no SteamVR (OpenVR, the serial bus and the compositor are stubbed). One test checks the SteamVR manifest the app generates and is skipped until you have connected VR once.
| File | Contents |
|---|---|
| docs/VR_QUICKSTART.md | Zero to both arms following your hands: setup order, controller layout, tuning, a long troubleshooting table |
| docs/MANUAL.md | Servo tool manual: every tab, every register, per-joint midpoint calibration, safety rules |
| docs/EE_MODE.md | End-effector (IK) mode: link lengths, zero pose, motor signs, acceptance test, tuning table, guards |
| docs/TELEOP_MATH.md | The complete mathematics of the pipeline, with a claim → test map |
| docs/TELEOP_IK_PLAN.md | The joint-space → task-space design plan the code implements |
| docs/report_draft.md | Plain-English overview of the system, results and lessons learned |
| docs/log.md | Development log: every field session as symptom → root cause → fix → decision, plus current state and open items |
Working on the author's robot as of September 2026: both arms follow both hands in EE mode, hold-to-move gripper, tank drive, camera in the headset with head-following gimbal, clean VR connect / disconnect / reconnect.
Open items (details in docs/log.md): tape-measure acceptance of the kinematic model (FK vs ruler < 1 cm at four poses), a servo overload cutoff monitor during teleop, an "EE home" pre-engage button, a twist → base-rotation channel, a shared config-write helper.
Known limitations: Windows only (developed and tested on Windows 11); the Rift S is the only headset tested (other SteamVR controllers get an auto-adapted binding file, untested); one operator at a time, since the Rift S tracks the controllers with the headset's cameras; servo firmware upgrade is deliberately not implemented (the bootloader protocol is undocumented, use the factory FD tool).
- xLeRobot — the robot. Its own VR stack, XLeVR, targets a standalone Quest over Wi-Fi; this tool reads a tethered headset directly through OpenVR, with clutch and calibration.
- SO-ARM100 / SO-101 — the arm; link lengths come from its URDF.
- lerobot — the ecosystem xLeRobot builds on.
- Feetech STS3215 servos and the factory FD debug tool this replaces.
MIT — see LICENSE. The bundled Montserrat font is licensed under the SIL Open Font License 1.1 (fonts/OFL.txt); it is loaded privately by the process at runtime and never installed system-wide.