Readd sim sglang

.agents/skills/cookbooks/cosmos3/simulation/README.md

@@ -8,10 +8,10 @@ spatial control video.
 
 | Model | Generates | From | Start here | Notebooks |
 | --- | --- | --- | --- | --- |
-| [Cosmos3-Nano-Sim-Bimanual](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Bimanual) | Video rollout of bimanual robot manipulation | Initial RGB frame, task description, time-aligned action trajectory | [Bimanual guide](bimanual/bimanual.md) | [Framework](bimanual/bimanual.md), [vLLM-Omni](bimanual/run_bimanual_with_vllm_omni.ipynb) |
-| [Cosmos3-Nano-Transfer-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto) | Synchronized multi-camera driving video, joint mode adds LiDAR | Per-camera WSM controls and captions, HD-map LiDAR controls for joint mode | [Auto overview](auto/README.md), [Transfer-Auto guide](auto/auto.md) | [Framework](auto/run_auto_with_cosmos_framework.ipynb), [vLLM-Omni](auto/run_auto_with_vllm_omni.ipynb) |
-| [Cosmos3-Nano-Sim-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Auto) | Same outputs as Transfer-Auto, generated chunk by chunk | Same inputs as Transfer-Auto | [Auto overview](auto/README.md) | [Framework](auto/run_sim-auto_with_cosmos_framework.ipynb), [vLLM-Omni](auto/run_sim_auto_with_vllm_omni.ipynb) |
-| [Cosmos3-Nano-Sim-Transfer](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Transfer) | 480p video | Caption and one edge, blur, depth, or segmentation control video | [Sim-Transfer guide](transfer/transfer.md) | [Framework](transfer/transfer.md), [vLLM-Omni](transfer/run_transfer_with_vllm_omni.ipynb) |
+| [Cosmos3-Nano-Sim-Bimanual](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Bimanual) | Video rollout of bimanual robot manipulation | Initial RGB frame, task description, time-aligned action trajectory | [Bimanual guide](bimanual/bimanual.md) | [Framework](bimanual/bimanual.md), [vLLM-Omni](bimanual/run_bimanual_with_vllm_omni.ipynb), [SGLang](bimanual/run_bimanual_with_sglang.ipynb) |
+| [Cosmos3-Nano-Transfer-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto) | Synchronized multi-camera driving video, joint mode adds LiDAR | Per-camera WSM controls and captions, HD-map LiDAR controls for joint mode | [Auto overview](auto/README.md), [Transfer-Auto guide](auto/auto.md) | [Framework](auto/run_auto_with_cosmos_framework.ipynb), [vLLM-Omni](auto/run_auto_with_vllm_omni.ipynb), [SGLang](auto/run_auto_with_sglang.ipynb) |
+| [Cosmos3-Nano-Sim-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Auto) | Same outputs as Transfer-Auto, generated chunk by chunk | Same inputs as Transfer-Auto | [Auto overview](auto/README.md) | [Framework](auto/run_sim-auto_with_cosmos_framework.ipynb), [vLLM-Omni](auto/run_sim_auto_with_vllm_omni.ipynb), [SGLang](auto/run_sim-auto_with_sglang.ipynb) |
+| [Cosmos3-Nano-Sim-Transfer](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Transfer) | 480p video | Caption and one edge, blur, depth, or segmentation control video | [Sim-Transfer guide](transfer/transfer.md) | [Framework](transfer/transfer.md), [vLLM-Omni](transfer/run_transfer_with_vllm_omni.ipynb), [SGLang](transfer/run_transfer_with_sglang.ipynb) |
 
 To drive the Auto models with your own scene, render WSM controls with
 [generate_wsm.ipynb](auto/generate_wsm.ipynb).
@@ -37,6 +37,11 @@ distilled student: a fixed four-step sampler with a rolling key/value cache.
   Bimanual and Sim-Transfer add model-specific setup steps in their guides.
 - **vLLM-Omni:** Docker with the NVIDIA Container Toolkit. All four notebooks use
   the `vllm/vllm-omni:cosmos3` image.
+- **SGLang:** Docker with the NVIDIA Container Toolkit. The four SGLang notebooks use
+  the published `lmsysorg/sglang:dev` image; the Auto and Sim-Transfer notebooks read
+  their inputs from the mounted cookbook root. The `dev` tag moves with SGLang's main
+  branch; the notebooks were validated against SGLang commit [`ab580b2e`](https://github.com/sgl-project/sglang/commit/ab580b2e060059c0f8387b598a9f648dcec13f93),
+  published as the fixed tag `lmsysorg/sglang:nightly-dev-20261020-ab580b2e`.
 
 If you pass `--defaults-file` to Framework inference, it **replaces** the built-in
 sampling defaults rather than merging with them. Start from the defaults file the

.agents/skills/cookbooks/cosmos3/simulation/auto/README.md

@@ -11,7 +11,9 @@ controls. Joint generation also needs numerical HD-map LiDAR controls.
 | Native Transfer-Auto | [Framework](run_auto_with_cosmos_framework.ipynb) | Seven cameras, 297 frames, 720p; optional LiDAR |
 | Native Sim-Auto | [Framework](run_sim-auto_with_cosmos_framework.ipynb) | Seven cameras, 297 frames, 720p; optional LiDAR |
 | Served Transfer-Auto | [vLLM-Omni](run_auto_with_vllm_omni.ipynb) | Seven cameras, 201 frames, 480p; WSM-only and joint examples |
+| Served Transfer-Auto | [SGLang](run_auto_with_sglang.ipynb) | Seven cameras, 201 frames, 480p; WSM-only and joint examples from the published `lmsysorg/sglang:dev` image |
 | Served Sim-Auto | [vLLM-Omni](run_sim_auto_with_vllm_omni.ipynb) | Seven cameras, 297 frames, 480p; RGB and joint examples |
+| Served Sim-Auto | [SGLang](run_sim-auto_with_sglang.ipynb) | Seven cameras, 297 frames, 480p; RGB example from the published `lmsysorg/sglang:dev` image |
 | Render your controls | [WSM renderer](generate_wsm.ipynb) | Controls from your own RDS-HQ scene and calibration |
 | Sim-Auto with FlashDreams | [FlashDreams](https://github.com/NVIDIA/flashdreams/integrations_v2/cosmos3_nano_sim_auto) | Integrate with [FlashDreams](https://flashdreams.org/main/models/cosmos3_nano_sim_auto/) for accelerated inference |
 

.agents/skills/cookbooks/cosmos3/simulation/auto/auto.md

@@ -16,12 +16,17 @@ model), and [Cosmos3-Nano-Sim-Transfer](../transfer/transfer.md).
 
 ## Supported backends
 
-This cookbook supports two runtimes:
+This cookbook supports three runtimes:
 
 - **Cosmos Framework** for export, native PyTorch inference.
 - **vLLM-Omni** for an OpenAI-compatible video server. The runnable
   [Multiview Simulation notebook](run_auto_with_vllm_omni.ipynb) uses the published WSM controls and the synchronous video endpoint, followed by a joint WSM/RGB/LiDAR example
   that retrieves both generated RGB and numeric LiDAR through asynchronous jobs.
+- **SGLang** for asynchronous WSM and joint camera/LiDAR generation. Its
+  [Multiview Simulation notebook](run_auto_with_sglang.ipynb) uses server-visible input paths
+  and downloads generated camera videos and numeric LiDAR through the Videos API. The
+  [Sim-Auto SGLang notebook](run_sim-auto_with_sglang.ipynb) runs the causal student's
+  seven-camera sample through the same API.
 
 These examples consume the same fixed camera order and WSM-per-view contract. The commands,
 request fields, and environment are runtime-specific.
@@ -295,6 +300,19 @@ you submit generation requests. The complete client is in
 
 ### Send the multipart request
 
+For SGLang generation, use [`run_auto_with_sglang.ipynb`](run_auto_with_sglang.ipynb).
+Call `prepare_request(..., backend="sglang")` to preserve absolute `control_path` and `vision_path` values
+and return multipart form fields with an empty upload list. Both backends encode
+`extra_params` as a JSON string. The control files must be readable
+at the same paths on the server; mount the checkout and prepared outputs into the server
+container at those paths. SGLang does not accept per-camera `control_reference_index` fields.
+Then call `submit_wsm(..., async_mode=True, timeout=3600)`. The helper submits multipart form data to `POST /v1/videos`,
+polls `GET /v1/videos/{id}`, and downloads `/content` when the job completes. It saves the
+latest status in `job.json` and raises on failure, cancellation, or polling timeout.
+For joint LiDAR, use `submit_joint(..., backend="sglang")` and the
+[SGLang LiDAR contract](#joint-rgb-and-lidar-with-sglang) below. The helpers default to the
+vLLM-Omni multipart contract and synchronous WSM endpoint shown below.
+
 `POST /v1/videos/sync` accepts one control per camera as repeated multipart `input_references`
 parts. In the JSON-encoded `extra_params`, each ordered view replaces its local `control_path`
 with a zero-based `control_reference_index` pointing at the corresponding upload. Each view
@@ -394,6 +412,62 @@ metadata against the job descriptor. It displays the camera grid and saves
 `sample_outputs.json`. Generated LiDAR remains in the downloaded safetensors file with its
 metadata sidecar for further analysis.
 
+### Joint RGB and LiDAR with SGLang
+
+The LiDAR section of [`run_auto_with_sglang.ipynb`](run_auto_with_sglang.ipynb) uses the same
+daytime sample and [joint request](assets/daytime_scene/transfer_auto_joint.json). `prepare_request(..., backend="sglang")` retains each camera's absolute
+`control_path` and `vision_path`, retains the numeric `lidar.control_path`, and translates the
+manifest's `lidar.return_output` to SGLang's `lidar.decode`. It returns an empty upload list;
+the server reads the files directly. Mount the checkout and prepared outputs at the same
+absolute paths inside the server container. The joint checkpoint must include `lidar_vae/`;
+the published `lmsysorg/sglang:dev` image runs the LiDAR VAE on its built-in FlexAttention kernel.
+
+With generated LiDAR enabled, the JSON-encoded `extra_params` contains:
+
+```json
+{
+  "lidar": {
+    "control_path": "/absolute/path/to/cosmos-framework/.agents/skills/cookbooks/cosmos3/simulation/outputs/sglang/auto/lidar/prepared/lidar_control.safetensors",
+    "decode": true
+  }
+}
+```
+
+The joint preparation cell converts the sample's `control.tar` into the FP32 tensor
+`frames[3,100,128,1800]` under the output folder. SGLang selects its first
+67 sweeps for 201 camera frames at 30 FPS and a 10 Hz LiDAR checkpoint, and handles padding
+and normalization. `submit_joint(..., backend="sglang")` submits `POST /v1/videos`, polls the
+job, and downloads the camera video from `/content` and numeric LiDAR from
+`/content?variant=rangemap`. Setting the manifest's `return_output=false` produces
+`decode=false`: LiDAR still conditions camera generation, but no numeric artifact is returned.
+The notebook keeps output enabled for its validation cells.
+
+SGLang's generated safetensors contains three tensors, each shaped `[67,128,1800]`:
+
+| Tensor | Dtype | Meaning |
+| --- | --- | --- |
+| `range_m` | FP32 | Range in metres; invalid rays are zero |
+| `intensity` | FP16 | Intensity in [0,1]; invalid rays are zero |
+| `validity` | uint8 | Binary validity mask |
+
+`validate_sglang_lidar()` checks these tensors and embedded metadata. The notebook compares
+them with the job descriptor's `sweeps`, `fps`, `height`, `width`, `min_range_m`, and
+`max_range_m`. SGLang's generated file uses a different layout from the input's `frames` tensor.
+
+Default paths below are relative to `.agents/skills/cookbooks/cosmos3/simulation/`:
+
+| Artifact | Path |
+| --- | --- |
+| Input LiDAR | `outputs/sglang/auto/lidar/prepared/lidar_control.safetensors` (converted locally) |
+| Generated LiDAR | `outputs/sglang/auto/lidar/multiview_camera_major.lidar.safetensors` |
+| Generated descriptor | `outputs/sglang/auto/lidar/multiview_camera_major.lidar.json` |
+
+The notebook prints the resolved input and generated paths. Its server URL defaults to
+`SGLANG_BASE_URL` (`http://127.0.0.1:30000`); optional authentication uses `SGLANG_API_KEY`.
+`COSMOS3_OUTPUT_DIR` and `AUTO_SAMPLE_ROOT` select the output folder and a local sample copy, as in
+the vLLM-Omni notebook. Because the server opens inputs by path, the notebook stages each RGB
+condition under the output folder; keep that folder inside the mounted cookbook root.
+
 ### vLLM-Omni troubleshooting
 
 - **`missing views[0].prompt`:** supply a nonempty `prompt` in every camera view, even for