Repository navigation
Readd sim sglang - #301
Readd sim sglang#301
Conversation
Four notebooks serve the Simulation models through the published lmsysorg/sglang:dev image (sgl-project/sglang#39475): Sim-Bimanual, Sim-Transfer, Sim-Auto, and Transfer-Auto (WSM-only and joint camera plus LiDAR). SGLang opens inputs by server-visible path instead of uploads, so the notebooks stage their inputs under the output folder inside the mounted cookbook root and the server command mounts that root. - helpers/sim_auto_sglang_helpers.py builds the Sim-Auto multipart form; helpers/auto_helpers.py gains the SGLang request, asynchronous job, control staging, and LiDAR validation paths; bimanual/bimanual_helpers.py encodes the action payload within the 1 MiB form-field limit. - CPU tests cover the request contracts and the notebook flows against a mock server; the cookbook workflow lints the new helper. - The Simulation README, the Auto overview, and the Transfer-Auto guide link the notebooks. Migrated from nvidia-cosmos/cosmos-framework-private branch readd-sim-sglang (ede4d0e9) as one change relative to its base 84a653ac, since the public history was synced by squash in NVIDIA#292. Ported to this repository: Hugging Face revisions follow main as in NVIDIA#294, because the previously pinned commits no longer exist, and the README keeps main's Framework links beside the new SGLang ones. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Kedi Wu <kediw@nvidia.com>
…ervers The Sim-Auto and Transfer-Auto SGLang notebooks now say how to use several GPUs: add --num-gpus N --ulysses-degree N --sp-degree N to sglang serve. Both pipelines accept Ulysses sequence parallelism only, the published checkpoints have 8 key/value heads, so N is 2, 4, or 8, and the server never detects the GPU count on its own. Sim-Auto may combine it with --tp-size under the same divisibility rule; Transfer-Auto may add --enable-cfg-parallel for the two guidance branches. This also drops the outdated claim that Sim-Auto rejects sequence parallelism. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Kedi Wu <kediw@nvidia.com>
pengcuo
left a comment
There was a problem hiding this comment.
Reviewed the helpers, tests and the four notebooks at cell level. The helper changes are solid and well covered (the MockTransport tests in particular), but I cannot call this mergeable yet:
- No description. Please add: why these notebooks were removed before and what changed (I assume sgl-project/sglang#39475, merged 2026-10-18), which
lmsysorg/sglang:devimage digest or commit the four notebooks were run against, and end-to-end evidence per notebook (Auto WSM, Auto joint LiDAR including the rangemap validation cell, Sim-Auto, Bimanual, Sim-Transfer). The second commit documents multi-GPU Ulysses flags; were those run? - Moving image tag. All four notebooks tell readers to
docker pull lmsysorg/sglang:dev"before each start" because "the tag moves". A cookbook that cannot be reproduced next month is a support burden. Please pin the first release tag (or a digest) that contains #39475 as soon as one exists, and record the date and commit in the version table until then. The vLLM-Omni notebooks at least pin a dedicatedvllm/vllm-omni:cosmos3tag. - CI coverage.
validate-cookbooks.ymlonly lints and tests undersimulation/auto. The newbimanual/bimanual_helpers.py(354 lines of request and validation logic) has no lint or unit test in CI; the notebook lint step covers notebooks only. Add it to a ruff step at minimum, ideally with a small unit test forprepare_bimanual_input/encode_extra_paramslike the Auto helpers have. - CI for this fork PR has not run yet; a maintainer needs to approve the workflow before merge.
Non-blocking: the Sim-Auto SGLang notebook re-implements submit_video_job / download_video / probe_video in its "client helpers" cell while this PR adds _submit_video_job / _download to auto_helpers.py; importing them would remove one copy. The vLLM-Omni notebooks carry the same cell, so this is existing convention. Smaller things inline.
… add note about which commit the notebook is validated against
1-3 are addressed. ulysses 4 is fully tested for auto notebooks |
sim-bimanual run:
agibot_mixed_028_bulk_seed2026.mp4
sim-transfer runs:
blur.mp4
seg.mp4
edge.mp4
depth.mp4
sim-auto run:
vision_view00_camera_front_wide_120fov.mp4
vision_view06_camera_front_tele_30fov.mp4
vision_view05_camera_cross_left_120fov.mp4
vision_view04_camera_rear_left_70fov.mp4
vision_view03_camera_rear_tele_30fov.mp4
vision_view02_camera_rear_right_70fov.mp4
vision_view01_camera_cross_right_120fov.mp4
transfer-auto wsm:
vision_view00_camera_front_wide_120fov.mp4
vision_view06_camera_front_tele_30fov.mp4
vision_view05_camera_cross_left_120fov.mp4
vision_view04_camera_rear_left_70fov.mp4
vision_view03_camera_rear_tele_30fov.mp4
vision_view02_camera_rear_right_70fov.mp4
vision_view01_camera_cross_right_120fov.mp4
transfer-auto lidar:
transfer_auto_lidar.mp4
lidar_info: {'shape': [67, 128, 1800], 'dtypes': {'range_m': 'F32', 'intensity': 'F16', 'validity': 'U8'}, 'metadata': {'fps': '10.0', 'layout': '[sweeps, beams, azimuth]', 'min_range_m': '5.0', 'max_range_m': '100.0', 'invalid_range_m': '0.0'}}