From ed95104db2f4a6575a9c424e7aede86020c7ec9d Mon Sep 17 00:00:00 2001 From: Yiming Zhao Date: Wed, 21 Oct 2026 11:36:37 -0700 Subject: [PATCH 1/2] docs(reasoner): add Cosmos3-Edge to the TensorRT-LLM cookbook. Nano and Super stay on the pinned release:1.3.0rc22 image. Edge is served from a source build that includes Cosmos3EdgeForConditionalGeneration. Signed-off-by: Yiming Zhao --- .agents/skills/cookbooks/cosmos3/README.md | 31 +++-- .../cookbooks/cosmos3/reasoner/README.md | 12 +- .../reasoner/run_with_tensorrt_llm.ipynb | 131 ++++++++++++++---- 3 files changed, 137 insertions(+), 37 deletions(-) diff --git a/.agents/skills/cookbooks/cosmos3/README.md b/.agents/skills/cookbooks/cosmos3/README.md index bfa07955e..ba10ccd8a 100644 --- a/.agents/skills/cookbooks/cosmos3/README.md +++ b/.agents/skills/cookbooks/cosmos3/README.md @@ -388,14 +388,19 @@ supports this payload: poll `GET /v1/videos/{id}`, then download it from ## TensorRT-LLM Reasoner -OpenAI-compatible **reasoning** server for image and video understanding. Run -TensorRT-LLM in the prebuilt -[`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`](https://catalog.ngc.nvidia.com/orgs/nvidia/tensorrt-llm/containers/release) -container, or follow the -[TensorRT-LLM Generator source-build instructions](#tensorrt-llm-generator) to -install TensorRT-LLM from source. +OpenAI-compatible **reasoning** server for image and video understanding. Authenticate with Hugging Face before loading gated checkpoints. +**Cosmos3-Nano** and **Cosmos3-Super** run in the prebuilt +[`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`](https://catalog.ngc.nvidia.com/orgs/nvidia/tensorrt-llm/containers/release) +container. + +**Cosmos3-Edge** uses `Cosmos3EdgeForConditionalGeneration`. That release image +does not include the architecture. Install TensorRT-LLM from a checkout that +includes [PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307), following +the [TensorRT-LLM Generator source-build instructions](#tensorrt-llm-generator), +and run the Edge command below in that environment. + Install headless OpenCV in the environment that runs the server. Cosmos3 Reasoner uses it to process vision inputs: @@ -405,8 +410,7 @@ python -m pip install opencv-python-headless ### Start the server -Run one of these commands inside the release container or activated local -environment. +**Cosmos3-Nano** and **Cosmos3-Super** — run inside the release container above. **Cosmos3-Nano** (single GPU, port 8001): @@ -429,6 +433,17 @@ trtllm-serve nvidia/Cosmos3-Super \ --max_num_tokens 32768 ``` +**Cosmos3-Edge** (single GPU, port 8001) — run in the source environment above, +not in `release:1.3.0rc22`: + +```bash +CUDA_VISIBLE_DEVICES=0 \ +trtllm-serve nvidia/Cosmos3-Edge \ + --host 0.0.0.0 \ + --port 8001 \ + --max_num_tokens 32768 +``` + The server exposes `/health` and the OpenAI-compatible API at `http://localhost:8001/v1`. See the [Reasoner TensorRT-LLM notebook](reasoner/run_with_tensorrt_llm.ipynb) for image diff --git a/.agents/skills/cookbooks/cosmos3/reasoner/README.md b/.agents/skills/cookbooks/cosmos3/reasoner/README.md index e86219b58..74c60fd32 100644 --- a/.agents/skills/cookbooks/cosmos3/reasoner/README.md +++ b/.agents/skills/cookbooks/cosmos3/reasoner/README.md @@ -325,8 +325,10 @@ GPUs. Set up the environment and start the server with the shared [TensorRT-LLM Reasoner setup](../README.md#tensorrt-llm-reasoner). It covers the prebuilt release container or a local source installation, Hugging Face -authentication, and launch commands for **Cosmos3-Nano** on one GPU or -**Cosmos3-Super** on four GPUs. +authentication, and launch commands for **Cosmos3-Nano** or **Cosmos3-Edge** +on one GPU, or **Cosmos3-Super** on four GPUs. Nano and Super use the pinned +`release:1.3.0rc22` image. Edge uses a source build that includes +`Cosmos3EdgeForConditionalGeneration`. The [`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) notebook publishes the selected server on host port **8001** and waits for `/health` @@ -364,8 +366,10 @@ print(response.choices[0].message.content) ### Notebook walkthrough -[`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) launches either -**Cosmos3-Nano** or **Cosmos3-Super**, waits for the server to become healthy, +[`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) launches +**Cosmos3-Nano** or **Cosmos3-Super** from the pinned release image, or +**Cosmos3-Edge** from a source build that includes +`Cosmos3EdgeForConditionalGeneration`. It waits for the server to become healthy and walks through image and video examples covering detailed captioning, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT trajectories, driving scenes, diff --git a/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb b/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb index a0c07aae2..7b78f0c99 100644 --- a/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb +++ b/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb @@ -7,16 +7,17 @@ "source": [ "# Cosmos3 Reasoner with TensorRT-LLM\n", "\n", - "This notebook serves Cosmos3 Reasoner through an OpenAI-compatible TensorRT-LLM server using the prebuilt `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`\n", - "Docker image. It:\n", + "This notebook serves Cosmos3 Reasoner through an OpenAI-compatible TensorRT-LLM server. **Cosmos3-Nano** and **Cosmos3-Super** use the prebuilt `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`\n", + "Docker image. **Cosmos3-Edge** needs a TensorRT-LLM build that includes `Cosmos3EdgeForConditionalGeneration`\n", + "([PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)); that release image does not. It:\n", "\n", - "1. Configures the local assets, installs notebook dependencies, and pulls the server image.\n", - "2. Launches Cosmos3-Nano or Cosmos3-Super in Docker.\n", + "1. Configures the local assets, installs notebook dependencies, and pulls the server image for Nano or Super.\n", + "2. Launches Cosmos3-Nano or Cosmos3-Super in Docker, or Cosmos3-Edge from the source build.\n", "3. Sends image and video reasoning requests with the `openai` client.\n", "\n", - "The notebook can serve `nvidia/Cosmos3-Nano` on one GPU or `nvidia/Cosmos3-Super` across\n", - "4 GPUs with tensor parallelism. The query examples in the final section resolve the served\n", - "model dynamically.\n" + "The notebook can serve `nvidia/Cosmos3-Nano` on one GPU, `nvidia/Cosmos3-Super` across\n", + "4 GPUs with tensor parallelism, or `nvidia/Cosmos3-Edge` on one GPU. Every tier listens on host port **8001**.\n", + "The query examples in the final section resolve the served model dynamically.\n" ] }, { @@ -26,15 +27,17 @@ "source": [ "## 1. Setup\n", "\n", - "Serving runs entirely in Docker, so no local Python environment is required for the model.\n", + "Nano and Super run in Docker, so no local Python environment is required for those tiers.\n", "You need:\n", "\n", - "- Docker with the NVIDIA Container Toolkit (`--runtime nvidia` / `--gpus`).\n", - "- The `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`or newer image, which ships native Cosmos3 Reasoner support\n", - " (tensorrt-llm ≥ 1.3.0rc22). A cell below pulls it (~20 GB on first run).\n", + "- Docker with the NVIDIA Container Toolkit (`--runtime nvidia` / `--gpus`) for Nano and Super.\n", + "- The `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22` image for Nano and Super. A cell below pulls it (~20 GB on first run).\n", + " Do not use this image for Edge.\n", + "- For Edge, a TensorRT-LLM checkout that includes `Cosmos3EdgeForConditionalGeneration`\n", + " ([PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)), installed with the source-build steps in the shared cookbook.\n", "\n", "The Cosmos3-Super Reasoner checkpoint is gated on Hugging Face. If you choose Super,\n", - "authenticate on the host with a token that has access before launching Super. Both launch\n", + "authenticate on the host with a token that has access before launching Super. The Nano and Super launch\n", "cells mount `~/.cache/huggingface` into the container so downloaded weights are reused:\n", "\n", "```bash\n", @@ -119,9 +122,9 @@ "source": [ "## 2. Launch a TensorRT-LLM server\n", "\n", - "Choose either the Nano or Super tier below. The commands are mutually exclusive: both use\n", - "the container name `cosmos3-reasoner-trtllm` and publish the container's port 8000 to host\n", - "port **8001**, which matches the endpoint the query cells use.\n", + "Choose one tier below. The commands are mutually exclusive. Nano and Super use the container\n", + "name `cosmos3-reasoner-trtllm` and publish the container's port 8000 to host port **8001**.\n", + "Edge serves directly on host port **8001**. That is the endpoint the query cells use.\n", "\n", "The first start downloads the weights and compiles CUDA graphs and can take several\n", "minutes; the readiness cell below polls `/health` and waits automatically." @@ -216,6 +219,67 @@ "echo \"Cosmos3-Super starting in container '$CONTAINER' — run the readiness cell next.\"\n" ] }, + { + "cell_type": "markdown", + "id": "trtllm-edge-md", + "metadata": {}, + "source": [ + "### Cosmos3-Edge inference with TensorRT-LLM\n", + "\n", + "`Cosmos3-Edge` is the single-GPU Reasoner. Install TensorRT-LLM from a checkout that includes\n", + "`Cosmos3EdgeForConditionalGeneration`\n", + "([PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)), using the source-build steps in\n", + "the shared cookbook. Do not launch Edge with `release:1.3.0rc22`.\n", + "\n", + "From that environment:\n", + "\n", + "```bash\n", + "CUDA_VISIBLE_DEVICES=0 \\\n", + "trtllm-serve nvidia/Cosmos3-Edge \\\n", + " --host 0.0.0.0 \\\n", + " --port 8001 \\\n", + " --max_num_tokens 32768\n", + "```\n", + "\n", + "The cell below starts that command when `trtllm-serve` is on `PATH` and leaves it listening on\n", + "port 8001. Then run the readiness cell.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "trtllm-edge-code", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euo pipefail\n", + "if ! command -v trtllm-serve >/dev/null 2>&1; then\n", + " echo \"trtllm-serve is not on PATH.\"\n", + " echo \"Install TensorRT-LLM from a checkout that includes Cosmos3EdgeForConditionalGeneration\"\n", + " echo \"(https://github.com/NVIDIA/TensorRT-LLM/pull/20307). release:1.3.0rc22 does not.\"\n", + " echo \"Run this cell again from that environment.\"\n", + " exit 1\n", + "fi\n", + "\n", + "docker rm -f cosmos3-reasoner-trtllm >/dev/null 2>&1 || true\n", + "\n", + "if [[ -f /tmp/cosmos3-edge-reasoner-trtllm.pid ]]; then\n", + " old_pid=\"$(cat /tmp/cosmos3-edge-reasoner-trtllm.pid)\"\n", + " if kill -0 \"$old_pid\" 2>/dev/null; then\n", + " kill \"$old_pid\" 2>/dev/null || true\n", + " wait \"$old_pid\" 2>/dev/null || true\n", + " fi\n", + "fi\n", + "\n", + "python -m pip install --no-cache-dir opencv-python-headless\n", + "nohup env CUDA_VISIBLE_DEVICES=0 trtllm-serve nvidia/Cosmos3-Edge \\\n", + " --host 0.0.0.0 --port 8001 --max_num_tokens 32768 \\\n", + " > /tmp/cosmos3-edge-reasoner-trtllm.log 2>&1 &\n", + "echo $! > /tmp/cosmos3-edge-reasoner-trtllm.pid\n", + "echo \"Cosmos3-Edge starting on port 8001 — run the readiness cell next.\"\n" + ] + }, { "cell_type": "markdown", "id": "367d55e2-cf33-4a57-805a-3cebb8bafea3", @@ -223,7 +287,7 @@ "source": [ "### Wait for the server to be ready\n", "\n", - "Run this after launching either tier. It streams the container log and polls `/health`\n", + "Run this after launching one tier. It streams the server log and polls `/health`\n", "until the server is ready, allowing up to 30 minutes on the first run while the weights\n", "download and CUDA graphs compile." ] @@ -239,11 +303,19 @@ "set -uo pipefail\n", "\n", "CONTAINER=cosmos3-reasoner-trtllm\n", - "\n", - "echo \"Streaming server logs; waiting for http://127.0.0.1:8001/health ...\"\n", - "docker logs -f \"$CONTAINER\" 2>&1 &\n", - "LOGPID=$!\n", - "trap 'kill \"$LOGPID\" 2>/dev/null || true' EXIT\n", + "EDGE_LOG=/tmp/cosmos3-edge-reasoner-trtllm.log\n", + "EDGE_PID_FILE=/tmp/cosmos3-edge-reasoner-trtllm.pid\n", + "\n", + "echo \"Waiting for http://127.0.0.1:8001/health ...\"\n", + "if docker ps -q -f \"name=^${CONTAINER}$\" 2>/dev/null | grep -q .; then\n", + " docker logs -f \"$CONTAINER\" 2>&1 &\n", + " LOGPID=$!\n", + " trap 'kill \"$LOGPID\" 2>/dev/null || true' EXIT\n", + "elif [[ -f \"$EDGE_LOG\" ]]; then\n", + " tail -n +1 -f \"$EDGE_LOG\" &\n", + " LOGPID=$!\n", + " trap 'kill \"$LOGPID\" 2>/dev/null || true' EXIT\n", + "fi\n", "\n", "for _ in $(seq 1 1800); do\n", " if curl -fsS \"http://127.0.0.1:8001/health\" >/dev/null 2>&1; then\n", @@ -251,10 +323,19 @@ " echo \"TensorRT-LLM server is ready.\"\n", " exit 0\n", " fi\n", - " if ! docker ps -q -f \"name=^${CONTAINER}$\" | grep -q .; then\n", - " echo\n", - " echo \"Container exited early; inspect the logs above.\"\n", - " exit 1\n", + " if docker ps -aq -f \"name=^${CONTAINER}$\" 2>/dev/null | grep -q .; then\n", + " if ! docker ps -q -f \"name=^${CONTAINER}$\" 2>/dev/null | grep -q .; then\n", + " echo\n", + " echo \"Container exited early; inspect the logs above.\"\n", + " exit 1\n", + " fi\n", + " elif [[ -f \"$EDGE_PID_FILE\" ]]; then\n", + " edge_pid=\"$(cat \"$EDGE_PID_FILE\")\"\n", + " if ! kill -0 \"$edge_pid\" 2>/dev/null; then\n", + " echo\n", + " echo \"Edge server exited early; inspect $EDGE_LOG.\"\n", + " exit 1\n", + " fi\n", " fi\n", " sleep 2\n", "done\n", From 7afec9187de2a7002725072145b1de3b59cf1b86 Mon Sep 17 00:00:00 2001 From: Yiming Zhao Date: Fri, 23 Oct 2026 16:50:27 -0700 Subject: [PATCH 2/2] docs(reasoner): address TensorRT-LLM Edge cookbook review comments. Stop only a confirmed Edge server, shut that process down with the container, and skip the Nano/Super image pull when Docker is absent. Point the Edge source build at TensorRT-LLM main after 39a44fab. Signed-off-by: Yiming Zhao --- .agents/skills/cookbooks/cosmos3/README.md | 17 +- .../cookbooks/cosmos3/reasoner/README.md | 21 ++- .../reasoner/run_with_tensorrt_llm.ipynb | 172 ++++++++++++++---- 3 files changed, 164 insertions(+), 46 deletions(-) diff --git a/.agents/skills/cookbooks/cosmos3/README.md b/.agents/skills/cookbooks/cosmos3/README.md index ba10ccd8a..4aae4e3e4 100644 --- a/.agents/skills/cookbooks/cosmos3/README.md +++ b/.agents/skills/cookbooks/cosmos3/README.md @@ -395,14 +395,17 @@ Authenticate with Hugging Face before loading gated checkpoints. [`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`](https://catalog.ngc.nvidia.com/orgs/nvidia/tensorrt-llm/containers/release) container. -**Cosmos3-Edge** uses `Cosmos3EdgeForConditionalGeneration`. That release image -does not include the architecture. Install TensorRT-LLM from a checkout that -includes [PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307), following -the [TensorRT-LLM Generator source-build instructions](#tensorrt-llm-generator), +**Cosmos3-Edge** uses `Cosmos3EdgeForConditionalGeneration`. Release images +through `v1.4.0rc1`, including `release:1.3.0rc22`, do not include that +architecture. Install TensorRT-LLM from upstream `main` at or after +[`39a44fab`](https://github.com/NVIDIA/TensorRT-LLM/commit/39a44fab74f4cfc0d4d498be385e746fdd182872) +(the merge of [PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)), +following the +[TensorRT-LLM Generator source-build instructions](#tensorrt-llm-generator), and run the Edge command below in that environment. -Install headless OpenCV in the environment that runs the server. Cosmos3 -Reasoner uses it to process vision inputs: +Install headless OpenCV with the same interpreter that provides `trtllm-serve`. +Cosmos3 Reasoner uses it to process vision inputs: ```bash python -m pip install opencv-python-headless @@ -434,7 +437,7 @@ trtllm-serve nvidia/Cosmos3-Super \ ``` **Cosmos3-Edge** (single GPU, port 8001) — run in the source environment above, -not in `release:1.3.0rc22`: +not in a release image through `v1.4.0rc1`: ```bash CUDA_VISIBLE_DEVICES=0 \ diff --git a/.agents/skills/cookbooks/cosmos3/reasoner/README.md b/.agents/skills/cookbooks/cosmos3/reasoner/README.md index 74c60fd32..d818012c5 100644 --- a/.agents/skills/cookbooks/cosmos3/reasoner/README.md +++ b/.agents/skills/cookbooks/cosmos3/reasoner/README.md @@ -327,15 +327,18 @@ Set up the environment and start the server with the shared prebuilt release container or a local source installation, Hugging Face authentication, and launch commands for **Cosmos3-Nano** or **Cosmos3-Edge** on one GPU, or **Cosmos3-Super** on four GPUs. Nano and Super use the pinned -`release:1.3.0rc22` image. Edge uses a source build that includes -`Cosmos3EdgeForConditionalGeneration`. +`release:1.3.0rc22` image. Edge uses a source build of TensorRT-LLM `main` at +or after `39a44fab`, which includes `Cosmos3EdgeForConditionalGeneration`. +Release images through `v1.4.0rc1` do not. The [`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) notebook publishes the selected server on host port **8001** and waits for `/health` before running the request examples. Point standalone clients at `http://localhost:8001/v1`. -Once the server is ready, query it with the OpenAI client: +Once the server is ready, query it with the OpenAI client. This request matches +the Cosmos Framework Edge example: one sentence, at most 1024 tokens, and +sampling disabled. ```python from pathlib import Path @@ -353,11 +356,15 @@ response = client.chat.completions.create( "role": "user", "content": [ {"type": "image_url", "image_url": {"url": image_url}}, - {"type": "text", "text": "Caption the image in detail."}, + { + "type": "text", + "text": "Describe what is happening in this image in one sentence.", + }, ], } ], - max_tokens=4096, + max_tokens=1024, + temperature=0, seed=0, ) @@ -368,8 +375,8 @@ print(response.choices[0].message.content) [`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) launches **Cosmos3-Nano** or **Cosmos3-Super** from the pinned release image, or -**Cosmos3-Edge** from a source build that includes -`Cosmos3EdgeForConditionalGeneration`. It waits for the server to become healthy +**Cosmos3-Edge** from a TensorRT-LLM `main` build at or after `39a44fab`. +It waits for the server to become healthy and walks through image and video examples covering detailed captioning, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT trajectories, driving scenes, diff --git a/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb b/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb index 7b78f0c99..c2ff99306 100644 --- a/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb +++ b/.agents/skills/cookbooks/cosmos3/reasoner/run_with_tensorrt_llm.ipynb @@ -8,10 +8,10 @@ "# Cosmos3 Reasoner with TensorRT-LLM\n", "\n", "This notebook serves Cosmos3 Reasoner through an OpenAI-compatible TensorRT-LLM server. **Cosmos3-Nano** and **Cosmos3-Super** use the prebuilt `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`\n", - "Docker image. **Cosmos3-Edge** needs a TensorRT-LLM build that includes `Cosmos3EdgeForConditionalGeneration`\n", - "([PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)); that release image does not. It:\n", + "Docker image. **Cosmos3-Edge** needs TensorRT-LLM `main` at or after [`39a44fab`](https://github.com/NVIDIA/TensorRT-LLM/commit/39a44fab74f4cfc0d4d498be385e746fdd182872), which includes `Cosmos3EdgeForConditionalGeneration`\n", + "([PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)). Release images through `v1.4.0rc1` do not. It:\n", "\n", - "1. Configures the local assets, installs notebook dependencies, and pulls the server image for Nano or Super.\n", + "1. Configures the local assets and installs notebook dependencies. The image-pull cell is for Nano or Super and skips the pull when Docker is unavailable.\n", "2. Launches Cosmos3-Nano or Cosmos3-Super in Docker, or Cosmos3-Edge from the source build.\n", "3. Sends image and video reasoning requests with the `openai` client.\n", "\n", @@ -31,10 +31,10 @@ "You need:\n", "\n", "- Docker with the NVIDIA Container Toolkit (`--runtime nvidia` / `--gpus`) for Nano and Super.\n", - "- The `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22` image for Nano and Super. A cell below pulls it (~20 GB on first run).\n", - " Do not use this image for Edge.\n", - "- For Edge, a TensorRT-LLM checkout that includes `Cosmos3EdgeForConditionalGeneration`\n", - " ([PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)), installed with the source-build steps in the shared cookbook.\n", + "- The `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22` image for Nano and Super. The pull cell below downloads it (~20 GB on first run) and is Nano/Super-only.\n", + " It skips the pull when Docker is unavailable. Do not use this image for Edge.\n", + "- For Edge, TensorRT-LLM `main` at or after [`39a44fab`](https://github.com/NVIDIA/TensorRT-LLM/commit/39a44fab74f4cfc0d4d498be385e746fdd182872), installed with the source-build steps in the shared cookbook.\n", + " Release images through `v1.4.0rc1` do not include `Cosmos3EdgeForConditionalGeneration`.\n", "\n", "The Cosmos3-Super Reasoner checkpoint is gated on Hugging Face. If you choose Super,\n", "authenticate on the host with a token that has access before launching Super. The Nano and Super launch\n", @@ -42,7 +42,7 @@ "\n", "```bash\n", "hf auth login\n", - "```" + "```\n" ] }, { @@ -110,9 +110,15 @@ "source": [ "%%bash\n", "set -euo pipefail\n", - ": \"${COSMOS3_TRTLLM_IMAGE:?run the setup cell first}\"\n", "\n", - "docker pull \"$COSMOS3_TRTLLM_IMAGE\"" + "if ! command -v docker >/dev/null 2>&1 || ! docker info >/dev/null 2>&1; then\n", + " echo \"Docker is not available. Skipping the Nano/Super image pull.\"\n", + " echo \"This cell is only for Nano and Super. Edge serves from the source build and does not use this image.\"\n", + " exit 0\n", + "fi\n", + "\n", + ": \"${COSMOS3_TRTLLM_IMAGE:?run the setup cell first}\"\n", + "docker pull \"$COSMOS3_TRTLLM_IMAGE\"\n" ] }, { @@ -226,10 +232,11 @@ "source": [ "### Cosmos3-Edge inference with TensorRT-LLM\n", "\n", - "`Cosmos3-Edge` is the single-GPU Reasoner. Install TensorRT-LLM from a checkout that includes\n", - "`Cosmos3EdgeForConditionalGeneration`\n", + "`Cosmos3-Edge` is the single-GPU Reasoner. Install TensorRT-LLM from upstream `main` at or after\n", + "[`39a44fab`](https://github.com/NVIDIA/TensorRT-LLM/commit/39a44fab74f4cfc0d4d498be385e746fdd182872)\n", "([PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)), using the source-build steps in\n", - "the shared cookbook. Do not launch Edge with `release:1.3.0rc22`.\n", + "the shared cookbook. Release images through `v1.4.0rc1`, including `release:1.3.0rc22`, do not\n", + "include `Cosmos3EdgeForConditionalGeneration`.\n", "\n", "From that environment:\n", "\n", @@ -241,8 +248,9 @@ " --max_num_tokens 32768\n", "```\n", "\n", - "The cell below starts that command when `trtllm-serve` is on `PATH` and leaves it listening on\n", - "port 8001. Then run the readiness cell.\n" + "The cell below starts that command when `trtllm-serve` is on `PATH`. It stops a previous Edge\n", + "server only when the recorded PID still belongs to `trtllm-serve nvidia/Cosmos3-Edge`, waits\n", + "until that process exits, and refuses to start while port 8001 is in use. Then run the readiness cell.\n" ] }, { @@ -254,30 +262,89 @@ "source": [ "%%bash\n", "set -euo pipefail\n", + "\n", + "PID_FILE=/tmp/cosmos3-edge-reasoner-trtllm.pid\n", + "LOG_FILE=/tmp/cosmos3-edge-reasoner-trtllm.log\n", + "PORT=8001\n", + "\n", + "edge_server_pid() {\n", + " local pid=\"$1\"\n", + " [[ \"$pid\" =~ ^[0-9]+$ ]] || return 1\n", + " [[ -r \"/proc/${pid}/cmdline\" ]] || return 1\n", + " local cmd\n", + " cmd=\"$(tr '\\0' ' ' < \"/proc/${pid}/cmdline\")\"\n", + " [[ \"$cmd\" == *trtllm-serve* && \"$cmd\" == *nvidia/Cosmos3-Edge* ]]\n", + "}\n", + "\n", + "port_in_use() {\n", + " local hex\n", + " hex=\"$(printf '%04X' \"$PORT\")\"\n", + " awk -v p=\"$hex\" 'NR > 1 { split($2, a, \":\"); if (a[2] == p && $4 == \"0A\") found = 1 } END { exit !found }' /proc/net/tcp 2>/dev/null && return 0\n", + " awk -v p=\"$hex\" 'NR > 1 { split($2, a, \":\"); if (a[2] == p && $4 == \"0A\") found = 1 } END { exit !found }' /proc/net/tcp6 2>/dev/null && return 0\n", + " return 1\n", + "}\n", + "\n", + "stop_previous_edge() {\n", + " [[ -f \"$PID_FILE\" ]] || return 0\n", + " local pid\n", + " pid=\"$(tr -d '[:space:]' < \"$PID_FILE\")\"\n", + " if ! edge_server_pid \"$pid\"; then\n", + " echo \"PID file ${PID_FILE} does not point at trtllm-serve nvidia/Cosmos3-Edge (pid ${pid}). Leaving that process alone.\"\n", + " rm -f \"$PID_FILE\"\n", + " return 0\n", + " fi\n", + " echo \"Stopping previous Cosmos3-Edge server pid ${pid}.\"\n", + " kill \"$pid\" 2>/dev/null || true\n", + " local _\n", + " for _ in $(seq 1 60); do\n", + " if ! kill -0 \"$pid\" 2>/dev/null; then\n", + " rm -f \"$PID_FILE\"\n", + " return 0\n", + " fi\n", + " sleep 1\n", + " done\n", + " echo \"Previous Cosmos3-Edge server pid ${pid} did not exit.\"\n", + " exit 1\n", + "}\n", + "\n", "if ! command -v trtllm-serve >/dev/null 2>&1; then\n", " echo \"trtllm-serve is not on PATH.\"\n", - " echo \"Install TensorRT-LLM from a checkout that includes Cosmos3EdgeForConditionalGeneration\"\n", - " echo \"(https://github.com/NVIDIA/TensorRT-LLM/pull/20307). release:1.3.0rc22 does not.\"\n", - " echo \"Run this cell again from that environment.\"\n", + " echo \"Install TensorRT-LLM from upstream main at or after 39a44fab\"\n", + " echo \"(https://github.com/NVIDIA/TensorRT-LLM/commit/39a44fab74f4cfc0d4d498be385e746fdd182872).\"\n", + " echo \"Release images through v1.4.0rc1 do not include Cosmos3EdgeForConditionalGeneration.\"\n", " exit 1\n", "fi\n", "\n", - "docker rm -f cosmos3-reasoner-trtllm >/dev/null 2>&1 || true\n", + "if command -v docker >/dev/null 2>&1; then\n", + " docker rm -f cosmos3-reasoner-trtllm >/dev/null 2>&1 || true\n", + "fi\n", + "\n", + "stop_previous_edge\n", "\n", - "if [[ -f /tmp/cosmos3-edge-reasoner-trtllm.pid ]]; then\n", - " old_pid=\"$(cat /tmp/cosmos3-edge-reasoner-trtllm.pid)\"\n", - " if kill -0 \"$old_pid\" 2>/dev/null; then\n", - " kill \"$old_pid\" 2>/dev/null || true\n", - " wait \"$old_pid\" 2>/dev/null || true\n", + "for _ in $(seq 1 50); do\n", + " if ! port_in_use; then\n", + " break\n", " fi\n", + " sleep 0.2\n", + "done\n", + "if port_in_use; then\n", + " echo \"Port ${PORT} is still in use. Stop the process listening there before launching Edge.\"\n", + " exit 1\n", "fi\n", "\n", - "python -m pip install --no-cache-dir opencv-python-headless\n", - "nohup env CUDA_VISIBLE_DEVICES=0 trtllm-serve nvidia/Cosmos3-Edge \\\n", - " --host 0.0.0.0 --port 8001 --max_num_tokens 32768 \\\n", - " > /tmp/cosmos3-edge-reasoner-trtllm.log 2>&1 &\n", - "echo $! > /tmp/cosmos3-edge-reasoner-trtllm.pid\n", - "echo \"Cosmos3-Edge starting on port 8001 — run the readiness cell next.\"\n" + "SERVE_BIN=\"$(command -v trtllm-serve)\"\n", + "SERVE_PY=\"$(dirname \"$SERVE_BIN\")/python\"\n", + "if [[ ! -x \"$SERVE_PY\" ]]; then\n", + " echo \"Could not find the Python interpreter next to trtllm-serve: ${SERVE_PY}\"\n", + " exit 1\n", + "fi\n", + "\"$SERVE_PY\" -m pip install --no-cache-dir opencv-python-headless\n", + "\n", + "nohup env CUDA_VISIBLE_DEVICES=0 \"$SERVE_BIN\" nvidia/Cosmos3-Edge \\\n", + " --host 0.0.0.0 --port \"$PORT\" --max_num_tokens 32768 \\\n", + " > \"$LOG_FILE\" 2>&1 &\n", + "echo $! > \"$PID_FILE\"\n", + "echo \"Cosmos3-Edge starting on port ${PORT} — run the readiness cell next.\"\n" ] }, { @@ -1386,7 +1453,7 @@ "source": [ "## 4. Shut down the server\n", "\n", - "Remove the detached server container when you are finished.\n" + "Stop whichever server this notebook started. Nano and Super run in the `cosmos3-reasoner-trtllm` container. Edge runs as a local `trtllm-serve` process. This cell removes the container when Docker is available, and stops the Edge process only when the recorded PID still belongs to `trtllm-serve nvidia/Cosmos3-Edge`. It waits for that process to exit.\n" ] }, { @@ -1399,7 +1466,48 @@ "%%bash\n", "set -euo pipefail\n", "\n", - "docker rm -f cosmos3-reasoner-trtllm\n" + "PID_FILE=/tmp/cosmos3-edge-reasoner-trtllm.pid\n", + "\n", + "edge_server_pid() {\n", + " local pid=\"$1\"\n", + " [[ \"$pid\" =~ ^[0-9]+$ ]] || return 1\n", + " [[ -r \"/proc/${pid}/cmdline\" ]] || return 1\n", + " local cmd\n", + " cmd=\"$(tr '\\0' ' ' < \"/proc/${pid}/cmdline\")\"\n", + " [[ \"$cmd\" == *trtllm-serve* && \"$cmd\" == *nvidia/Cosmos3-Edge* ]]\n", + "}\n", + "\n", + "if command -v docker >/dev/null 2>&1; then\n", + " docker rm -f cosmos3-reasoner-trtllm >/dev/null 2>&1 || true\n", + " echo \"Removed container cosmos3-reasoner-trtllm if it was running.\"\n", + "else\n", + " echo \"Docker is not available. Skipped the Nano/Super container shutdown.\"\n", + "fi\n", + "\n", + "if [[ ! -f \"$PID_FILE\" ]]; then\n", + " echo \"No Cosmos3-Edge PID file.\"\n", + " exit 0\n", + "fi\n", + "\n", + "pid=\"$(tr -d '[:space:]' < \"$PID_FILE\")\"\n", + "if ! edge_server_pid \"$pid\"; then\n", + " echo \"PID file does not point at trtllm-serve nvidia/Cosmos3-Edge (pid ${pid}). Left that process running.\"\n", + " rm -f \"$PID_FILE\"\n", + " exit 0\n", + "fi\n", + "\n", + "echo \"Stopping Cosmos3-Edge server pid ${pid}.\"\n", + "kill \"$pid\" 2>/dev/null || true\n", + "for _ in $(seq 1 60); do\n", + " if ! kill -0 \"$pid\" 2>/dev/null; then\n", + " rm -f \"$PID_FILE\"\n", + " echo \"Cosmos3-Edge server stopped.\"\n", + " exit 0\n", + " fi\n", + " sleep 1\n", + "done\n", + "echo \"Cosmos3-Edge server pid ${pid} did not exit.\"\n", + "exit 1\n" ] } ],