Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 28 additions & 10 deletions .agents/skills/cookbooks/cosmos3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -388,25 +388,32 @@ supports this payload: poll `GET /v1/videos/{id}`, then download it from

## TensorRT-LLM Reasoner

OpenAI-compatible **reasoning** server for image and video understanding. Run
TensorRT-LLM in the prebuilt
[`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`](https://catalog.ngc.nvidia.com/orgs/nvidia/tensorrt-llm/containers/release)
container, or follow the
[TensorRT-LLM Generator source-build instructions](#tensorrt-llm-generator) to
install TensorRT-LLM from source.
OpenAI-compatible **reasoning** server for image and video understanding.
Authenticate with Hugging Face before loading gated checkpoints.

Install headless OpenCV in the environment that runs the server. Cosmos3
Reasoner uses it to process vision inputs:
**Cosmos3-Nano** and **Cosmos3-Super** run in the prebuilt
[`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22`](https://catalog.ngc.nvidia.com/orgs/nvidia/tensorrt-llm/containers/release)
container.

**Cosmos3-Edge** uses `Cosmos3EdgeForConditionalGeneration`. Release images
through `v1.4.0rc1`, including `release:1.3.0rc22`, do not include that
architecture. Install TensorRT-LLM from upstream `main` at or after
[`39a44fab`](https://github.com/NVIDIA/TensorRT-LLM/commit/39a44fab74f4cfc0d4d498be385e746fdd182872)
(the merge of [PR 20307](https://github.com/NVIDIA/TensorRT-LLM/pull/20307)),
following the
[TensorRT-LLM Generator source-build instructions](#tensorrt-llm-generator),
and run the Edge command below in that environment.

Install headless OpenCV with the same interpreter that provides `trtllm-serve`.
Cosmos3 Reasoner uses it to process vision inputs:

```bash
python -m pip install opencv-python-headless
```

### Start the server

Run one of these commands inside the release container or activated local
environment.
**Cosmos3-Nano** and **Cosmos3-Super** — run inside the release container above.

**Cosmos3-Nano** (single GPU, port 8001):

Expand All @@ -429,6 +436,17 @@ trtllm-serve nvidia/Cosmos3-Super \
--max_num_tokens 32768
```

**Cosmos3-Edge** (single GPU, port 8001) — run in the source environment above,
not in a release image through `v1.4.0rc1`:

```bash
CUDA_VISIBLE_DEVICES=0 \
trtllm-serve nvidia/Cosmos3-Edge \
--host 0.0.0.0 \
--port 8001 \
--max_num_tokens 32768
```

The server exposes `/health` and the OpenAI-compatible API at
`http://localhost:8001/v1`. See the
[Reasoner TensorRT-LLM notebook](reasoner/run_with_tensorrt_llm.ipynb) for image
Expand Down
25 changes: 18 additions & 7 deletions .agents/skills/cookbooks/cosmos3/reasoner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,15 +325,20 @@ GPUs.
Set up the environment and start the server with the shared
[TensorRT-LLM Reasoner setup](../README.md#tensorrt-llm-reasoner). It covers the
prebuilt release container or a local source installation, Hugging Face
authentication, and launch commands for **Cosmos3-Nano** on one GPU or
**Cosmos3-Super** on four GPUs.
authentication, and launch commands for **Cosmos3-Nano** or **Cosmos3-Edge**
on one GPU, or **Cosmos3-Super** on four GPUs. Nano and Super use the pinned
`release:1.3.0rc22` image. Edge uses a source build of TensorRT-LLM `main` at
or after `39a44fab`, which includes `Cosmos3EdgeForConditionalGeneration`.
Release images through `v1.4.0rc1` do not.

The [`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) notebook
publishes the selected server on host port **8001** and waits for `/health`
before running the request examples. Point standalone clients at
`http://localhost:8001/v1`.

Once the server is ready, query it with the OpenAI client:
Once the server is ready, query it with the OpenAI client. This request matches
the Cosmos Framework Edge example: one sentence, at most 1024 tokens, and
sampling disabled.

```python
from pathlib import Path
Expand All @@ -351,11 +356,15 @@ response = client.chat.completions.create(
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": image_url}},
{"type": "text", "text": "Caption the image in detail."},
{
"type": "text",
"text": "Describe what is happening in this image in one sentence.",
},
],
}
],
max_tokens=4096,
max_tokens=1024,
temperature=0,
seed=0,
)

Expand All @@ -364,8 +373,10 @@ print(response.choices[0].message.content)

### Notebook walkthrough

[`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) launches either
**Cosmos3-Nano** or **Cosmos3-Super**, waits for the server to become healthy,
[`run_with_tensorrt_llm.ipynb`](./run_with_tensorrt_llm.ipynb) launches
**Cosmos3-Nano** or **Cosmos3-Super** from the pinned release image, or
**Cosmos3-Edge** from a TensorRT-LLM `main` build at or after `39a44fab`.
It waits for the server to become healthy
and walks through image and video examples covering detailed captioning,
temporal localization, embodied reasoning, common-sense reasoning, 2D
grounding, describe-anything, action CoT trajectories, driving scenes,
Expand Down
Loading
Loading