Compose policies, robot embodiments and inference strategies.
Run, visualize, record and evaluate experiments in one shared framework.
Quick Start · Policies · Inference · Viewer · Evaluation · Docs
A real-robot rollout in ManiMux: live camera views, robot visualization and an action-chunk timeline.
ManiMux is a unified real-robot experiment platform built around an asynchronous inference runtime. The policy decides what should happen next; ManiMux decides when and how to execute it smoothly and safely, then records the complete run for inspection and evaluation.
| Policy | Embodiment | Inference |
|---|---|---|
| Plug in a policy through a shared adapter contract | Map observations and actions through robot-specific adapters | Swap scheduling and sampling strategies while sharing the execution layer |
The goal is composability, not a separate deployment stack for every combination. Current hardware validation centers on dual YAM; see integrations and the algorithm roadmap for the status of individual combinations.
cameras + robot state
↓
Policy / XPolicy model predicts the next 16-50 action steps
↓
PolicyAdapter converts them into executable robot joints
↓
ManiMux Runtime + Strategy schedules inference, trims stale actions and checks safety
↓
Robot submits both arms atomically on one control tick
Side outputs: Recorder episode + 3D Viewer
Remember three rules:
- The model never commands the robot directly. It only returns a future action chunk.
- The adapter translates semantics. Joint, EE-pose and delta outputs become one canonical chunk.
- ManiMux executes. Control timing, Timeline, Safety and Recorder are not copied per model.
ManiMux is also a repeatable experiment surface, not only a deployment runtime:
| Capability | What it gives the operator |
|---|---|
| Config-composed experiments | Swap the policy, inference strategy and embodiment without rewriting the control loop |
| Viewer-controlled rollouts | Prepare, start, finish and home each trial from one guided interface |
| Live action-chunk timeline | See inference latency, stale-prefix trimming, RTC conditioning, chunk handoff and gripper events while the robot runs |
| Synchronized evidence | Save resolved config, observations, raw/committed actions, robot states, events and camera video under one rollout |
| Human evaluation mode | Attach task result, smoothness and failure tags to a finished rollout for later comparison |
Normal mode keeps the same Viewer workflow without requiring labels. Experiment mode requires an evaluation before advancing, so large real-robot studies do not silently lose human annotations.
git clone --recursive https://github.com/SII-LiuLab/manimux.git
cd manimux
uv sync --dev
uv run manimux run --config configs/mock.yamlconfigs/mock.yaml uses a fake robot, camera and policy while exercising the real worker,
Timeline, SmoothExecutor, SafetyGuard and Recorder. It runs 120 control ticks and writes one
episode under data/; it never touches a camera, CAN interface or physical arm.
Launch the independent YAM viewer demo:
uv run manimux-viewer --robot yam --demo --port 8086Open http://localhost:8086. The --demo process is a standalone visualization demo, not a live
view of the mock runtime above.
For the live rollout state machine, button meanings, experiment labels and recovery flow, open the Viewer visual tutorial.
Copy the config first so the repository baseline stays unchanged:
cp configs/mock.yaml /tmp/manimux-beginner.yamlChange one field at a time and rerun:
| Change | Suggested value | What it demonstrates |
|---|---|---|
run.max_steps |
120 → 300 |
Control ticks and the episode lifecycle |
policy.inference_delay_s |
0.04 → 0.20 |
The control loop does not block on a slower model |
execution.blend_steps |
0 / 2 / 8 |
The seam between consecutive action chunks |
Read the first config as seven blocks: run defines the experiment, robot the embodiment,
sensors the observation, policy the model and chunk, execution scheduling and commands,
viewer live display, and recording the recording policy (the current runtime always records).
See the
field-by-field config tour.
| Path | Think of it as |
|---|---|
configs/ |
Experiment entry points: model, embodiment, checkpoint and runtime |
docs/*-runbook.md |
Operating instructions for installing, starting, checking and stopping a model |
src/manimux/ |
Shared engine: runtime, robot, sensor, safety, recording and viewer |
XPolicyLab/ |
Model internals: adapters and samplers for Pi05, GR00T, XR-1 and LingBot |
data/ |
Run output: resolved config, actions, states, events and episode result |
Running experiments normally requires only configs/ and one runbook. Enter src/manimux/ to
change shared infrastructure, and enter XPolicyLab/ only for model preprocessing, flow denoising
or model-native RTC sampling.
Chunk-based policy inference is commonly slower than a robot control period. ManiMux moves inference out of the control loop and owns replaceable inference strategies, chunk scheduling, stale-prefix trimming, atomic dual-arm commits, execution constraints, safety checks, recording and live visualization.
ManiMux is not tied to one model library. MolmoAct and ABC have native service adapters;
additional foundation models enter through XPolicyLab/. ManiMux keeps the stable runtime and
wire bridge. Model preprocessing, flow denoising and model-native RTC sampling stay in the
XPolicyLab adapter; embodiment mapping and execution safety stay in ManiMux.
Two loops at different speeds. The whole design is about the hand-off between them.
%%{init: {"flowchart": {"wrappingWidth": 260, "curve": "basis"}}}%%
flowchart TB
OBS["<b>BUILD ONE OBSERVATION</b><br/>3 cameras + 14 joint angles<br/>which views · what size · what units<br/><br/>PolicyAdapter"]:::stage
QUEUE(["bounded queue<br/><b>newest wins</b>"]):::pipe
subgraph THINK["<b>THINK</b> — every ~0.5 s, in its own process, 120-600 ms — <b>swap the model here</b>"]
direction LR
MOLMO["<b>MolmoAct2</b><br/>native"]:::molmo
ABCM["<b>ABC</b><br/>native"]:::abc
XR1["<b>XR-1</b><br/>XPolicy"]:::xr1
PI05["<b>Pi05</b><br/>XPolicy"]:::pi
GROOT["<b>GR00T N1.7</b><br/>XPolicy"]:::groot
LING["<b>LingBot-VLA2</b><br/>XPolicy"]:::ling
MOLMO ~~~ ABCM ~~~ XR1 ~~~ PI05 ~~~ GROOT ~~~ LING
end
DEC["<b>MAKE IT EXECUTABLE<br/>ON THIS BODY</b><br/>joint order · units · gripper<br/><b>EE→joint IK</b><br/><br/>PolicyAdapter"]:::stage
TL["<b>PUT THE CHUNK<br/>ON THE CLOCK</b><br/>drop steps already past<br/>blend the seam<br/><b>swap BOTH arms at once</b><br/><br/>ActionTimeline"]:::handoff
ACT["<b>ACT</b> · every 10-50 ms<br/><b>never waits for anything</b><br/>where should the arms be now?<br/>cap speed and acceleration<br/>refuse out-of-limit commands<br/><br/>Smooth │ MPC · SafetyGuard"]:::stage
ROBOT(["<b>one command,<br/>both arms together</b><br/>RobotDriver"]):::robot
SIDE(["episode on disk<br/>live 3D viewer"]):::side
OBS --> QUEUE --> THINK
THINK -->|"a chunk:<br/>next 16-50 steps"| DEC
DEC -->|"canonical<br/>joint-space chunk"| TL
TL --> ACT --> ROBOT
ACT -.->|"best-effort"| SIDE
classDef stage fill:#F6F8FA,stroke:#8C959F,stroke-width:1px,color:#1F2328
classDef handoff fill:#FFF4E5,stroke:#E36209,stroke-width:2.5px,color:#1F2328
classDef pipe fill:#FFFFFF,stroke:#8C959F,stroke-dasharray:4 3,color:#57606A
classDef side fill:#FFFFFF,stroke:#8C959F,stroke-dasharray:4 3,color:#57606A
classDef robot fill:#1F2328,stroke:#1F2328,color:#FFFFFF
classDef molmo fill:#2F6FEB,stroke:#1B4DB1,color:#FFFFFF
classDef abc fill:#1A7F55,stroke:#125C3D,color:#FFFFFF
classDef xr1 fill:#8957E5,stroke:#6633B8,color:#FFFFFF
classDef pi fill:#E36209,stroke:#A8460A,color:#FFFFFF
classDef groot fill:#CF222E,stroke:#96101A,color:#FFFFFF
classDef ling fill:#9A6700,stroke:#6E4A00,color:#FFFFFF
style THINK fill:#FFFFFF,stroke:#8C959F,stroke-dasharray:5 4,color:#1F2328
The control loop never waits — not for the model, disk, viewer, or a log line. A stale or invalid chunk never reaches the robot — wrong session, old sequence, past deadline, bad shape, non-finite value, or a dual-arm plan the two arms disagree on drops the whole chunk with a logged reason.
The real-robot control loop exists only once. Its inference strategy is replaceable:
the default strategy refills when the Timeline runs low, while RTC derives a condition
and soft mask from real execution progress. XPolicy advertises sampler capabilities at
handshake time, so an RTC config fails before robot connection when the live model has no
get_action_rtc hook.
| Path | Responsibility |
|---|---|
src/manimux/runtime/ |
One control loop plus pluggable inference strategies, Timeline, Smooth/MPC and Safety |
src/manimux/policies/ |
PolicyModel / PolicyAdapter contracts and isolated worker |
src/manimux/integrations/ |
Native integrations and the shared XPolicy WebSocket bridge |
src/manimux/robots/ |
RobotDriver implementations; dual YAM is the current hardware target |
src/manimux/sensors/ |
RealSense and the shared camera service |
src/manimux/recording/ |
Episodes, Zarr, events and action lineage |
src/manimux/viewer/ |
Generic viewer protocol and robot geometry adapters |
XPolicyLab/ |
Submodule pointing to our XPolicyLab fork; model-internal changes live here |
checkpoints/pretrained/ |
Unmodified foundation-model or upstream release weights |
checkpoints/finetuned/<publisher>/ |
Fine-tuned checkpoints, named after their published repository |
configs/<model>/<embodiment>/ |
One explicit configuration per model and embodiment experiment |
docs/*-runbook.md |
Per-model environment, checkpoint, contract and default startup |
docs/reproductions/ |
Per-method equations, integration, commands, evidence and review records |
For example, the Robocurve YAM releases live at
checkpoints/finetuned/robocurve/{pi05-yam-molmoact2,gr00t-n1.7-yam-molmoact2};
the official OpenPI pi05_base remains under checkpoints/pretrained/.
Status: ✅ running · 🧪 experimental · 🚧 not deployable yet · 🔌 infrastructure
| Status | Model/path | ManiMux canonical action | Config | Runbook |
|---|---|---|---|---|
| ✅ | MolmoAct2 + YAM | 30 × 14 joint positions | configs/molmoact2/yam/ |
MolmoAct2 |
| ✅ | ABC + YAM | 30 × 14 joint positions | configs/abc/yam/ |
ABC |
| ✅ | OpenPI Pi05 + YAM | 16/50 × 14 absolute joint positions | configs/pi05/yam/ |
Pi05 |
| ✅ | GR00T N1.7 + YAM | 16 × 14 absolute joint positions | configs/groot/yam/ |
GR00T |
| ✅ | XR-1 + YAM | 30×60 EE delta → 30×14 joints |
configs/xiaomi-xr1/yam/ |
Runbook |
| ✅ | LingBot-VLA2 + YAM | 50×14 arm-relative + absolute gripper → joints |
configs/lingbot-vla2/yam/ |
Runbook |
| 🔌 | SAPolicy + YAM (XPolicyLab WS) | absolute dual-EE → YAM joints via sapolicy_yam |
configs/sapolicy/yam/ |
Runbook |
| 🧪 | Cosmos3 DROID | 32×8 single-arm absolute joints; offline only |
XPolicyLab/policy/Cosmos3/ |
Offline runbook |
| 🚧 | Isaac 0.5 LIBERO | 8×7 checkpoint-native absolute EE; model-only |
configs/isaac05/libero/ |
Offline runbook |
| 🔌 | XPolicy bridge | Observation/action wire contract | configs/xpolicylab/yam/ |
Runbook |
Here ✅ means the GPU/server, camera, adapter, scheduling, robot and Recorder path has been exercised; it does not claim task success or YAM post-training quality. Model-specific checkpoints, action contracts, known failures and evidence stay in the linked runbooks.
Status: ✅ hardware exercised · 🧪 integrated and offline verified · 🚧 integrating · 📋 planned · 👀 tracked but training-dependent. Completion means the method path has run on hardware; it does not mean that every policy succeeds at the task or that every method performs well. All methods in the main list run from an existing checkpoint without user retraining.
| Status | Algorithm | Integration layer | Current evidence | Method documentation | Upstream |
|---|---|---|---|---|---|
| ✅ | Default (ManiMux asynchronous chunk execution) | Runtime | Exercised across the current dual-YAM model integrations | Architecture | This repository |
| ✅ | RTC (inference-time Pi-guided) | Flow sampler + Runtime | Pi05/YAM hardware; measured d=3–5, no post-start chunk gap |
XPolicy RTC contract | Paper · Kinetix code |
| ✅ | ACT Temporal Ensembling | Runtime | Pi05 step-1000/YAM hardware; operator observed smooth continuous execution | Method | Pinned ACT source · LeRobot |
| ✅ | Adaptive Action Chunking (AAC) | Pi05/GR00T multi-sample + Runtime | Pi05 step-1000/YAM hardware; functional but visibly pauses at roughly 0.51 s warmed latency |
Core audit · Pi05 audit | GR00T server · official selector |
| ✅ | PAINT paper reproduction | Pi05 flow sampler + Runtime | Pi05 step-1000/YAM hardware; operator observed markedly improved continuity | Pi05 audit and commands | Paper · official repository currently contains documentation only |
| ✅ | AutoHorizon JAX port | Pi05 action-expert introspection + synchronous Runtime | Pi05/YAM hardware exercised; faithful synchronous cadence caused visible inference holds | Pi05 audit and commands | Code |
| 📋 | SGAC | Diffusion-policy sampler + Runtime | Official release targets low-dimensional Diffusion Policy; a Pi05 flow port would be a non-official extension | — | Code |
| ✅ | DVAC paper reproduction | Pi05 flow sampler + Runtime | Pi05/YAM hardware exercised; trajectory judged more accurate than the preceding method, with synchronous pauses | Pi05 audit | Paper · official code not located |
| 📋 | ProbeFlow | Flow solver acceleration | Denoising velocity probes | — | Paper · official code not located |
| 📋 | DiscreteRTC | Discrete-diffusion sampler | Requires a compatible pretrained masked-token policy | — | Code · StarVLA |
Update submodules in an existing checkout:
git submodule update --init --recursiveThe core development environment uses ./.venv, the YAM runtime uses envs/yam/.venv, and model
servers use isolated environments. Follow the environment guide and one model
runbook for CUDA, Torch, checkpoints, preflight and safe shutdown.
Start the two model-independent services once:
# Terminal 1: cameras
envs/yam/.venv/bin/manimux-camera-server --config configs/cameras.yaml# Terminal 2: Viewer
envs/yam/.venv/bin/manimux-viewer --robot yam --host 0.0.0.0 --port 8086Then choose exactly one model recipe. Each recipe starts its model server, performs one no-CAN forward/adapter probe, and starts the Viewer-controlled runtime. Do not run two model recipes or two ManiMux runtimes for the same robot simultaneously.
# Run every command from the repository root:
cd /home/ubuntu/manimux
# Terminal 3: policy server
XPolicyLab/policy/Pi_05/openpi/.venv/bin/python \
scripts/servers/pi05_yam_server.py \
--config configs/pi05/yam/server/finetune-assemble-screwdriver-step15000.yaml# Terminal 4: probe, then runtime service
envs/yam/.venv/bin/python scripts/validation/xpolicylab_yam_forward_probe.py \
--config configs/pi05/yam/infra/manimux-assemble-screwdriver-step15000.yaml
envs/yam/.venv/bin/manimux serve \
--config configs/pi05/yam/infra/manimux-assemble-screwdriver-step15000.yamlTo use Pi05's inference-time RTC strategy, keep the same policy server running and use the
RTC runtime configuration below. The server is started only once; do not start a second Pi05
server on port 8500.
# Terminal 4: optional no-CAN probe against the RTC contract
cd /home/ubuntu/manimux
envs/yam/.venv/bin/python scripts/validation/xpolicylab_yam_forward_probe.py \
--config configs/pi05/yam/infra/rtc-assemble-screwdriver-step15000.yaml
# Terminal 5: ManiMux + Pi-guided RTC runtime
envs/yam/.venv/bin/manimux serve \
--config configs/pi05/yam/infra/rtc-assemble-screwdriver-step15000.yamlThe ordinary Pi05 recipe uses manimux-assemble-screwdriver-step15000.yaml (runtime: manimux);
the RTC recipe uses rtc-assemble-screwdriver-step15000.yaml (runtime: rtc). Choose one
runtime for a rollout, never both at the same time.
# Terminal 3: policy server
bash XPolicyLab/policy/LingBot_VLA2/setup_eval_policy_server.sh \
configs/lingbot-vla2/yam/server/finetune-assemble-screwdriver-step15000.yaml# Terminal 4: probe, then runtime service
envs/yam/.venv/bin/python scripts/validation/xpolicylab_yam_forward_probe.py \
--config configs/lingbot-vla2/yam/infra/manimux-assemble-screwdriver-step15000.yaml
envs/yam/.venv/bin/manimux serve \
--config configs/lingbot-vla2/yam/infra/manimux-assemble-screwdriver-step15000.yamlFor sampler-level RTC, keep the same LingBot policy server and replace the runtime config:
envs/yam/.venv/bin/manimux serve \
--config configs/lingbot-vla2/yam/infra/rtc-assemble-screwdriver-step15000.yaml# Terminal 3: policy server
envs/xr1/.venv/bin/python scripts/servers/xiaomi_xr1_yam_server.py \
--config configs/xiaomi-xr1/yam/server/finetune-assemble-screwdriver-step15000.yaml# Terminal 4: probe, then runtime service
envs/yam/.venv/bin/python scripts/validation/xpolicylab_yam_forward_probe.py \
--config configs/xiaomi-xr1/yam/infra/manimux-assemble-screwdriver-step15000.yaml \
--instruction "Assemble the screwdriver."
envs/yam/.venv/bin/manimux serve \
--config configs/xiaomi-xr1/yam/infra/manimux-assemble-screwdriver-step15000.yamlFor sampler-level RTC, keep the same XR-1 policy server and replace the runtime config:
envs/yam/.venv/bin/manimux serve \
--config configs/xiaomi-xr1/yam/infra/rtc-assemble-screwdriver-step15000.yamlOpen http://localhost:8086, then operate Viewer in this order:
- Confirm the task command and, for experiments, enter a layout/condition ID.
- Click
Prepare normal rollout, or🧪 Prepare experiment rolloutwhen labels are required. - Wait for
Connected · PAUSED, then clickStart rollout. - Use
Pause / Holdonly when execution must stop without ending the rollout. - Click
Finish & Hometo stop, save the episode, and run the configured home path. - After an experiment rollout, select the result and smoothness score, then save the evaluation.
Viewer shows only the controls for the current stage. The main viewport keeps a top/left/right camera wall beside the robot digital twin, so live images do not require sidebar scrolling. A two-lane action timeline below the cameras shows inference in progress, the active chunk cursor, RTC-conditioned overlap, commit-time trimming, and the old tail replaced at each chunk switch.
These snippets show entry points only. They do not replace checkpoint validation, preflight, CAN checks or shutdown procedures. Model setup stays in its runbook; ACT, AAC, PAINT and later strategy commands stay in the linked method documentation in the algorithm list.
ManiMux includes PRM-as-a-Judge 1.5 as the PRM-as-a-Judge/ submodule. Our
recorded-video evaluations use Robo-Dopamine-GRM-2.0-8B-Preview as the judge.
This runs offline; it does not start a robot or a policy server.
The current entry point requires a JSONL manifest, not just a video root directory.
Save one JSON object per rollout in a file such as data/prm/manifest.jsonl:
{"case_id":"pi05-rollout-003","task":"Assemble the screwdriver.","video":"/absolute/path/to/session/rollout-003/videos/front_camera.mp4","model":"pi05_step15000","benchmark":"yam_real_top"}case_id, task (or instruction), and video are required. Relative video paths resolve
against the manifest's directory. Add more lines to evaluate multiple rollouts or models;
use distinct case IDs and set model for grouped comparisons. No human success label is required.
For our YAM top-view setup, select front_camera.mp4 only. The Dopamine adapter repeats that
single video into its three camera slots; this is top-only, not three-view evaluation.
From the repository root, with the dedicated prm-judge Conda environment installed:
git submodule update --init PRM-as-a-Judge
PRM_GPU_MEMORY_UTILIZATION=0.68 \
conda run --no-capture-output -n prm-judge python scripts/evaluation/prm_as_a_judge.py eval \
--manifest data/prm/manifest.jsonl \
--prm dopamine \
--prm-path checkpoints/pretrained/Robo-Dopamine-GRM-2.0-8B-Preview \
--output-root data/prm/results \
--gpus 0 --eval-mode forward --frame-interval 72 --batch-size 10 \
--outlier-method none --smoothing none --visualizeThe weights are local and excluded from Git. --frame-interval 72 means one sample every
72 source frames, not 72 samples per video. Results appear in data/prm/results/run_*/:
run_summary.json, per_case.jsonl, metrics.xlsx, report.md, and visualizations/report.html.
The toolkit computes progress/process metrics including MC@25/50/75, MP, PPL, CRA, STR and DRR;
you can report these without making success rate the headline result.
For environment setup, input views and report interpretation, see the
PRM usage guide.
configs/
<model>/
<embodiment>/
server/ # model-service configs
<checkpoint>.yaml
infra/ # ManiMux robot, sensors, execution and recording
<experiment>.yaml
cameras.yaml
robots/
mock.yaml
One experiment uses one explicit config. Different checkpoints, runtimes and embodiments do not share hidden switches. See configs/README.md for naming rules.
| Boundary | Responsibility |
|---|---|
PolicyModel |
Call a local model or model service; never touch the robot |
PolicyAdapter |
Camera/state encoding, action decoding, normalization, joint mapping and FK/IK |
RobotDriver |
Vendor SDK, state, commands, home, stop and close |
SensorDriver |
Read named, timestamped sensor streams |
RobotAdapter |
Viewer URDF, FK, joint groups and appearance |
Changing a model only changes the two Policy layers. Changing a robot only changes the embodiment adapter, driver and config. Runtime, Timeline, Safety, Recorder and the viewer protocol are not copied per model. See Architecture for the complete contracts.
- PRM-as-a-Judge offline video evaluation
- Real-robot experiment design
- Experiment infrastructure and Viewer workflow
- Viewer visual tutorial
- XPolicyLab integration
- MolmoAct2 · ABC
- Pi05 · GR00T · XPolicy XR-1
- LingBot-VLA2 · SAPolicy · CAN bus
- YAM 三模型训练流程
- Cosmos3 offline · Isaac 0.5 offline · ManiUniCon simulation
- Architecture
ManiMux drives physical robots. Passing tests does not prove hardware safety, and software cannot replace a physical emergency stop, hardware limits or the vendor watchdog. The runtime starts paused, faults do not auto-recover, and every hardware entry point requires its runbook preflight.
See THIRD_PARTY_NOTICES.md and licenses/ for upstream
attribution.
