1- <p align =" center " >
2- <a href =" README_ZH.md " >简体中文</a > | <strong >English</strong >
3- </p >
4-
51# SimplerEnv WidowX Bridge Eval
62
7- This directory runs StarVLA on the SimplerEnv WidowX Bridge tasks. It can run
8- the original Python ` .pt ` checkpoint and the robot.cpp CUDA GGUF build with the
9- same rollout settings, then compare their results.
3+ This directory evaluates the robot.cpp StarVLA GGUF runtime on the SimplerEnv
4+ WidowX Bridge tasks.
105
116## Protocol
127
13- - Four Bridge tasks, object episodes ` 0..23 ` , at most 120 steps, and 5 Hz
14- control.
15- - Official visual-matching RGB overlay resized to a 224x224 ` image_0 ` with
16- OpenCV ` INTER_AREA ` .
17- - A new action chunk each step and the official seven-prediction FP32 adaptive
18- ensemble.
19- - The Python and C++ runs use the same task, episode, seed, prompt, image,
20- action postprocessing, and termination conditions.
21- - Diffusion variants receive the same BF16-rounded initial noise for every
22- episode and policy step through protocol v4.
23-
24- The full comparison contains 96 rollouts per runtime: 4 tasks x 24 object
25- episodes. A smaller run is marked ` partial ` . The comparison reports success
26- rates but does not assign a pass threshold. Action differences can be checked
27- separately with ` compare_starvla_actions.py ` , whose default relative-L2 limit is
28- 3%.
29-
30- ## Setup
8+ - Four Bridge tasks with object episodes ` 0..23 `
9+ - At most 120 steps per episode at 5 Hz
10+ - Visual-matching RGB overlay resized to 224x224 with OpenCV ` INTER_AREA `
11+ - One action chunk per step with the official seven-prediction adaptive ensemble
3112
32- First download and convert the checkpoint as described in the
33- [ StarVLA conversion guide] ( ../../tools/hf2gguf/starvla/README.md ) , and build the
34- CUDA runtime. Paired evaluation also needs the catalog-pinned StarVLA checkout
35- and a local Python reference environment:
13+ A full run contains 96 rollouts. The result reports overall and per-task success
14+ rates; subset runs use ` partial ` coverage.
3615
37- ``` bash
38- STARVLA_REV=" $( python -c ' import json; print(json.load(open("tools/hf2gguf/starvla/checkpoint_catalog.json"))["source_revisions"]["starvla"])' ) "
39- git clone https://github.com/starVLA/starVLA.git ckpts/starvla/source/starvla
40- git -C ckpts/starvla/source/starvla checkout " ${STARVLA_REV} "
41-
42- python3.11 -m venv ckpts/starvla/.venv-official
43- ckpts/starvla/.venv-official/bin/pip install \
44- -c tools/hf2gguf/starvla/reference_constraints.txt \
45- -r ckpts/starvla/source/starvla/requirements.txt
46- ckpts/starvla/.venv-official/bin/pip install -e ckpts/starvla/source/starvla
47- ```
16+ ## Setup
4817
49- Override these locations with ` CHECKPOINT_ROOT ` , ` STARVLA_SOURCE ` ,
50- ` REFERENCE_PYTHON ` , or ` SIMPLER_ENV_ROOT ` . The FAST reference also requires the
51- retained ` ckpts/starvla/work/qwen25_fast ` staging directory.
18+ Convert a checkpoint as described in the
19+ [ StarVLA guide] ( ../../tools/hf2gguf/starvla/README.md ) and build the CUDA runtime.
5220
53- SimplerEnv uses these pinned revisions:
21+ The environment uses these revisions:
5422
5523``` text
56- SimplerEnv: 06accaca93535902d408da4855f21cece12bceb7
57- ManiSkill2_real2sim: ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682
24+ SimplerEnv: 06accaca93535902d408da4855f21cece12bceb7
25+ ManiSkill2_real2sim: ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682
5826```
5927
6028``` bash
@@ -71,65 +39,35 @@ pip install -e ckpts/simpler_env/source/SimplerEnv/ManiSkill2_real2sim
7139pip install -e ckpts/simpler_env/source/SimplerEnv
7240```
7341
74- Headless simulation needs a working Vulkan ICD. Validate environment creation
75- before starting rollouts:
76-
77- ``` bash
78- python ckpts/starvla/source/starvla/examples/simBenchmarks/SimplerEnv/eval_files/test_your_simplerEnv.py
79- ```
80-
81- The paired runner uses the system Vulkan loader by default. Set
82- ` VULKAN_LIBRARY_PATH ` and ` VULKAN_ICD ` only when a custom loader directory or
83- ICD JSON file is required.
42+ Headless simulation requires a working Vulkan ICD. Run SimplerEnv's environment
43+ test first to confirm that SAPIEN can find a rendering device.
8444
8545## Run
8646
8747` VARIANT ` accepts ` oft ` , ` groot ` , ` pi_v3 ` , ` qwen25_oft ` , ` qwen25_groot ` ,
88- ` qwen25_pi ` , and ` qwen25_fast ` . The scripts select the checkpoint, Qwen files,
89- GGUF files, Python server, and normalization profile for the chosen variant.
48+ ` qwen25_pi ` , and ` qwen25_fast ` .
9049
91- Inspect the full command without running it:
92-
93- ``` bash
94- VARIANT=oft DRY_RUN=1 \
95- bash eval/simpler_env/scripts/run_paired_local.sh
96- ```
97-
98- Run all 96 rollouts on four GPUs:
50+ Run the full profile:
9951
10052``` bash
53+ CUDA_VISIBLE_DEVICES=0 \
10154VARIANT=oft \
102- COMPARISON_ID=oft-local-paired-001 \
103- GPU_IDS=0,1,2,3 PORTS=5600,5601,5602,5603 \
104- SIMPLER_ENV_ROOT=ckpts/simpler_env/source/SimplerEnv \
105- bash eval/simpler_env/scripts/run_paired_local.sh
106- ```
107-
108- The runner completes four Python shards, then four C++ shards with the same
109- task/GPU/port mapping, and writes:
110-
111- ``` text
112- ckpts/starvla/results/<variant>/bridge-local-paired-<comparison-id>/comparison.json
55+ OUTPUT=ckpts/starvla/results/oft/bridge.json \
56+ bash eval/simpler_env/scripts/run_model_server.sh
11357```
11458
115- An explicit smoke subset must allow partial coverage :
59+ Run one smoke episode :
11660
11761``` bash
118- VARIANT=groot EPISODE_IDS=0 ALLOW_PARTIAL=1 \
119- COMPARISON_ID =groot-smoke-001 \
120- bash eval/simpler_env/scripts/run_paired_local .sh
62+ CUDA_VISIBLE_DEVICES=0 \
63+ VARIANT =groot TASK_IDS=0 EPISODE_IDS=0 \
64+ bash eval/simpler_env/scripts/run_model_server .sh
12165```
12266
123- The result is marked ` status=partial ` . Use ` run_python_reference.sh ` for a
124- standalone Python shard and ` run_model_server.sh ` for a standalone C++ shard or
125- a full run.
126-
127- ## Output
128-
129- Each ` (task, repeat) ` gets a fresh backend server. Results record the StarVLA
130- ` variant ` , checkpoint, action shape, and one row per rollout. Before running,
131- the C++ script verifies the conversion manifest and the GGUF file hashes. The
132- comparison rejects results produced with different models or rollout settings.
67+ The script reads three GGUF files from ` ckpts/starvla/gguf/<variant> ` and uses
68+ ` build_cuda/bin/model-server ` by default. Common overrides are ` GGUF_DIR ` ,
69+ ` SERVER_BIN ` , ` PYTHON ` , ` SIMPLER_ENV_ROOT ` , ` UNNORM_KEY ` , ` TASK_IDS ` ,
70+ ` EPISODE_IDS ` , ` REPEATS ` , and ` OUTPUT ` .
13371
134- ` comparison.json ` contains the success counts and rates, percentage-point
135- difference, per-episode agreement, and contingency table .
72+ Each task/repeat starts a fresh model-server. Results include checkpoint
73+ identity, rollout records, success rates, and timing summaries .
0 commit comments