Skip to content

Readd sim sglang - #301

Merged
lfengad merged 5 commits into
NVIDIA:mainfrom
kediwu0331:readd-sim-sglang
Oct 23, 2026
Merged

lfengad merged 5 commits into
NVIDIA:mainfrom
kediwu0331:readd-sim-sglang

Conversation

@kediwu0331

@kediwu0331 kediwu0331 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

sim-bimanual run:

agibot_mixed_028_bulk_seed2026.mp4

sim-transfer runs:

blur.mp4
seg.mp4
edge.mp4
depth.mp4

sim-auto run:

vision_view00_camera_front_wide_120fov.mp4
vision_view06_camera_front_tele_30fov.mp4
vision_view05_camera_cross_left_120fov.mp4
vision_view04_camera_rear_left_70fov.mp4
vision_view03_camera_rear_tele_30fov.mp4
vision_view02_camera_rear_right_70fov.mp4
vision_view01_camera_cross_right_120fov.mp4

transfer-auto wsm:

vision_view00_camera_front_wide_120fov.mp4
vision_view06_camera_front_tele_30fov.mp4
vision_view05_camera_cross_left_120fov.mp4
vision_view04_camera_rear_left_70fov.mp4
vision_view03_camera_rear_tele_30fov.mp4
vision_view02_camera_rear_right_70fov.mp4
vision_view01_camera_cross_right_120fov.mp4

transfer-auto lidar:

transfer_auto_lidar.mp4

lidar_info: {'shape': [67, 128, 1800], 'dtypes': {'range_m': 'F32', 'intensity': 'F16', 'validity': 'U8'}, 'metadata': {'fps': '10.0', 'layout': '[sweeps, beams, azimuth]', 'min_range_m': '5.0', 'max_range_m': '100.0', 'invalid_range_m': '0.0'}}

nv-kediw and others added 2 commits October 21, 2026 12:19
Four notebooks serve the Simulation models through the published
lmsysorg/sglang:dev image (sgl-project/sglang#39475): Sim-Bimanual,
Sim-Transfer, Sim-Auto, and Transfer-Auto (WSM-only and joint camera plus
LiDAR). SGLang opens inputs by server-visible path instead of uploads, so
the notebooks stage their inputs under the output folder inside the
mounted cookbook root and the server command mounts that root.

- helpers/sim_auto_sglang_helpers.py builds the Sim-Auto multipart form;
  helpers/auto_helpers.py gains the SGLang request, asynchronous job,
  control staging, and LiDAR validation paths; bimanual/bimanual_helpers.py
  encodes the action payload within the 1 MiB form-field limit.
- CPU tests cover the request contracts and the notebook flows against a
  mock server; the cookbook workflow lints the new helper.
- The Simulation README, the Auto overview, and the Transfer-Auto guide
  link the notebooks.

Migrated from nvidia-cosmos/cosmos-framework-private branch
readd-sim-sglang (ede4d0e9) as one change relative to its base 84a653ac,
since the public history was synced by squash in NVIDIA#292. Ported to this
repository: Hugging Face revisions follow main as in NVIDIA#294, because the
previously pinned commits no longer exist, and the README keeps main's
Framework links beside the new SGLang ones.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Kedi Wu <kediw@nvidia.com>
…ervers

The Sim-Auto and Transfer-Auto SGLang notebooks now say how to use
several GPUs: add --num-gpus N --ulysses-degree N --sp-degree N to
sglang serve. Both pipelines accept Ulysses sequence parallelism only,
the published checkpoints have 8 key/value heads, so N is 2, 4, or 8,
and the server never detects the GPU count on its own. Sim-Auto may
combine it with --tp-size under the same divisibility rule; Transfer-Auto
may add --enable-cfg-parallel for the two guidance branches. This also
drops the outdated claim that Sim-Auto rejects sequence parallelism.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Kedi Wu <kediw@nvidia.com>

@pengcuo pengcuo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the helpers, tests and the four notebooks at cell level. The helper changes are solid and well covered (the MockTransport tests in particular), but I cannot call this mergeable yet:

  1. No description. Please add: why these notebooks were removed before and what changed (I assume sgl-project/sglang#39475, merged 2026-10-18), which lmsysorg/sglang:dev image digest or commit the four notebooks were run against, and end-to-end evidence per notebook (Auto WSM, Auto joint LiDAR including the rangemap validation cell, Sim-Auto, Bimanual, Sim-Transfer). The second commit documents multi-GPU Ulysses flags; were those run?
  2. Moving image tag. All four notebooks tell readers to docker pull lmsysorg/sglang:dev "before each start" because "the tag moves". A cookbook that cannot be reproduced next month is a support burden. Please pin the first release tag (or a digest) that contains #39475 as soon as one exists, and record the date and commit in the version table until then. The vLLM-Omni notebooks at least pin a dedicated vllm/vllm-omni:cosmos3 tag.
  3. CI coverage. validate-cookbooks.yml only lints and tests under simulation/auto. The new bimanual/bimanual_helpers.py (354 lines of request and validation logic) has no lint or unit test in CI; the notebook lint step covers notebooks only. Add it to a ruff step at minimum, ideally with a small unit test for prepare_bimanual_input / encode_extra_params like the Auto helpers have.
  4. CI for this fork PR has not run yet; a maintainer needs to approve the workflow before merge.

Non-blocking: the Sim-Auto SGLang notebook re-implements submit_video_job / download_video / probe_video in its "client helpers" cell while this PR adds _submit_video_job / _download to auto_helpers.py; importing them would remove one copy. The vLLM-Omni notebooks carry the same cell, so this is existing convention. Smaller things inline.

Comment thread .agents/skills/cookbooks/cosmos3/simulation/README.md
Comment thread .github/workflows/validate-cookbooks.yml
@kediwu0331

Copy link
Copy Markdown
Contributor Author

Reviewed the helpers, tests and the four notebooks at cell level. The helper changes are solid and well covered (the MockTransport tests in particular), but I cannot call this mergeable yet:

  1. No description. Please add: why these notebooks were removed before and what changed (I assume sgl-project/sglang#39475, merged 2026-10-18), which lmsysorg/sglang:dev image digest or commit the four notebooks were run against, and end-to-end evidence per notebook (Auto WSM, Auto joint LiDAR including the rangemap validation cell, Sim-Auto, Bimanual, Sim-Transfer). The second commit documents multi-GPU Ulysses flags; were those run?
  2. Moving image tag. All four notebooks tell readers to docker pull lmsysorg/sglang:dev "before each start" because "the tag moves". A cookbook that cannot be reproduced next month is a support burden. Please pin the first release tag (or a digest) that contains #39475 as soon as one exists, and record the date and commit in the version table until then. The vLLM-Omni notebooks at least pin a dedicated vllm/vllm-omni:cosmos3 tag.
  3. CI coverage. validate-cookbooks.yml only lints and tests under simulation/auto. The new bimanual/bimanual_helpers.py (354 lines of request and validation logic) has no lint or unit test in CI; the notebook lint step covers notebooks only. Add it to a ruff step at minimum, ideally with a small unit test for prepare_bimanual_input / encode_extra_params like the Auto helpers have.
  4. CI for this fork PR has not run yet; a maintainer needs to approve the workflow before merge.

Non-blocking: the Sim-Auto SGLang notebook re-implements submit_video_job / download_video / probe_video in its "client helpers" cell while this PR adds _submit_video_job / _download to auto_helpers.py; importing them would remove one copy. The vLLM-Omni notebooks carry the same cell, so this is existing convention. Smaller things inline.

1-3 are addressed. ulysses 4 is fully tested for auto notebooks

@lfengad
lfengad merged commit 867b107 into NVIDIA:main Oct 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants