[ConstBob]

Summary

Moves the Cosmos3-Edge TensorRT-Edge-LLM user guide from NVIDIA/cosmos#326 into this repo, under .agents/skills/cookbooks/cosmos3/.

• Shared setup in .agents/skills/cookbooks/cosmos3/README.md, pinned to TensorRT-Edge-LLM v0.11.0. EDGELLM_ROOT defaults to $HOME/TensorRT-Edge-LLM.
• Reasoner notebook: export nvidia/Cosmos3-Edge with --task reasoning, build engines under ENGINE_DIR/reasoning, and run llm_inference on assets/robot_153.jpg.
• Policy notebook: on v0.11.0, build und-prefill, gen, and vae-encoder with tensorrt-edgellm-build under ENGINE_DIR/policy, then run cosmos3_policy_inference. The C++ runtime needs -DBUILD_EXPERIMENTAL_MODELS=ON and -DENABLE_CUTE_DSL=ALL. It uses the same 640×540 DROID concat_view as the vLLM-Omni / SGLang policy notebooks, with --viewPoint concat_view and an 8-value --state. Edge-LLM covers the policy task only.
• Notebooks resolve assets from .agents/skills/cookbooks/cosmos3.

On the v0.11.0 tag, tensorrt-edgellm-export --task policy writes the UND input attention_pos_id and VAE height / width only. cosmos3_policy_inference binds attention_position_ids and reads builder_config.content_height / content_width. The direct builder writes that contract, so the policy cookbook uses tensorrt-edgellm-build.

Upstream references: Cosmos3-Edge VLA guide for reasoning, and the direct engine builder for policy.

Testing

• Asset paths resolve from both notebook directories: robot_153.jpg and the three DROID camera videos are found.
• Policy, pinned v0.11.0 (95515c2), x86 RTX PRO 6000 (SM120): tensorrt-edgellm-build --components und-prefill,gen,vae-encoder, then cosmos3_policy_inference --viewPoint concat_view --state 0,0,0,0,0,0,0,0. action.json is [1, 32, 8], finite=true. The zero state is a runtime smoke, not a DROID quality check. v0.11.0 cosmos3_policy_inference --help lists concat_view.
• Reasoner was not re-run on v0.11.0. The last full export → llm_build + visual_build → llm_inference sequence was TensorRT-Edge-LLM v0.10.0 on the same RTX PRO 6000. After Ever's visual-export fix, llm_inference was re-run on that box for robot_153.jpg and describe_anything.png, and the captions matched the images. Ever confirmed normal outputs on Thor. See the thread on cookbook: add Cosmos3-Edge TensorRT-Edge-LLM reasoner and policy notebooks cosmos#326. That retest did not name the tag as v0.11.0.

[pengcuo]

Thanks for moving this over and for being explicit about the v0.11.0 state in the description. That is also why I am requesting changes rather than approving:

1. The documented policy path is known not to run. Both notebooks and the shared README pin TensorRT-Edge-LLM v0.11.0, but per your own testing the v0.11.0 cosmos3_policy_inference rejects engines produced by the documented tensorrt-edgellm-export --task policy -> cosmos3_policy_build path (missing attention_position_ids on und_prefill, content_height/width on vae_encoder), and the only end-to-end smoke was on v0.10.0 engines. A cookbook whose main path fails on the version it pins should not merge until either (a) the upstream export/build writes what the runtime needs and the notebook has been re-run on that version, or (b) the cookbook pins the combination that was actually exercised (v0.10.0 export, build and inference) and says so. The open question in your description ("waiting on which build path this cookbook should document") is the blocker; happy to re-review once it is settled.
2. Please state in the PR which version and box the reasoner notebook's export -> build -> llm_inference sequence was last run on; the description only references caption quality confirmed on Thor in the old thread.

Non-blocking:

• --viewPoint concat_view is passed to cosmos3_policy_inference. The upstream VLA guide does not mention the flag; experimental_models/cosmos3/examples/cosmos3_policy_inference.cpp on upstream main has it, so please confirm it exists in the pinned tag.
• README default EDGELLM_ROOT="${EDGELLM_ROOT:-$PWD/TensorRT-Edge-LLM}" vs the notebooks' ~/TensorRT-Edge-LLM; pick one.
• The policy engines are built directly under ENGINE_DIR while the reasoner uses ENGINE_DIR/reasoning; a policy/ subfolder keeps the two from sharing a directory (the notebook already puts its WORK_DIR under ENGINE_DIR/policy).

[pengcuo]

Per the PR description, v0.11.0's cosmos3_policy_inference does not load engines produced by the v0.11.0 tensorrt-edgellm-export --task policy + cosmos3_policy_build path documented here, and the only end-to-end smoke was on v0.10.0. Pin what was actually run, or re-run on the version that is pinned, before this merges.

[ConstBob]

The policy cookbook now uses tensorrt-edgellm-build on the pinned v0.11.0 tag (f19c188), instead of tensorrt-edgellm-export --task policy followed by cosmos3_policy_build. On that tag the export writes the UND input attention_pos_id and VAE height / width only, while cosmos3_policy_inference binds attention_position_ids and reads builder_config.content_height / content_width. The direct builder writes that contract.

I re-ran the documented build and inference on v0.11.0 (95515c2) on an x86 RTX PRO 6000 (SM120): tensorrt-edgellm-build --components und-prefill,gen,vae-encoder, then cosmos3_policy_inference --viewPoint concat_view --state 0,0,0,0,0,0,0,0. action.json is [1, 32, 8], finite=true. The zero state is a runtime smoke, not a DROID quality check. The PR testing section now records this run.

[pengcuo]

Default here is $PWD/TensorRT-Edge-LLM; both notebooks default to ~/TensorRT-Edge-LLM. One default, please.

[ConstBob]

Updated in f19c188. The shared README now defaults to $HOME/TensorRT-Edge-LLM, matching both notebooks (Path.home() / "TensorRT-Edge-LLM").

[pengcuo]

The upstream VLA guide (fetched 2026-10-21) does not list --viewPoint; upstream main has it in cosmos3_policy_inference.cpp. Please confirm the pinned tag accepts it, otherwise this cell fails on the version the notebook names.

[ConstBob]

Confirmed on the pinned v0.11.0 binary. cosmos3_policy_inference --help lists concat_view (with ego_view, third_person_view, and wrist_view). The v0.11.0 smoke passed --viewPoint concat_view and completed. The notebook now states that this tag accepts the flag.

[nvluxiaoz]

Thanks. Please ping latest version (0.11.0) to ensure it works and merge. Thanks a lot!

[nvluxiaoz]

LGTM

[ConstBob]

Hi @nvluxiaoz @ever-wong, following up on the v0.11.0 policy build-path question in the previous PR: NVIDIA/cosmos#326 (comment)

The documented VLA path is tensorrt-edgellm-export --task policy followed by cosmos3_policy_build. On the v0.11.0 tag, that export writes the UND input attention_pos_id, while cosmos3_policy_inference binds attention_position_ids. make_vae_encoder_config also writes only height, width, and num_frames, and cosmos3_policy_build copies that config through. The runtime requires vae_encoder builder_config.content_height and content_width.

tensorrt-edgellm-build on the same tag writes attention_position_ids and content_height / content_width. Its documentation still marks that frontend as experimental and says the supported workflow remains the ONNX export path. I have not re-exported or rebuilt engines on v0.11.0 yet.

Which path should this cookbook use?

1. Keep the documented VLA export path until the export and runtime field names match.
2. Switch the policy cookbook to tensorrt-edgellm-build for v0.11.0.

Thanks!

[ConstBob]

Thanks for the review. I confirmed with Luxiao that this policy cookbook should use the direct builder, tensorrt-edgellm-build, for v0.11.0.

1. The policy path in f19c188 is now tensorrt-edgellm-build, not tensorrt-edgellm-export --task policy followed by cosmos3_policy_build. On the pinned tag the export writes the UND input attention_pos_id and VAE height / width only, while cosmos3_policy_inference binds attention_position_ids and reads builder_config.content_height / content_width. The direct builder writes that contract. I re-ran that build and cosmos3_policy_inference on v0.11.0 (95515c2) on an x86 RTX PRO 6000 (SM120), with --viewPoint concat_view and --state 0,0,0,0,0,0,0,0. action.json is [1, 32, 8], finite=true. The zero state is a runtime smoke, not a DROID quality check. The PR testing section now records this run.

2. The reasoner notebook was not re-run on v0.11.0. The last full export → llm_build + visual_build → llm_inference sequence was TensorRT-Edge-LLM v0.10.0 on the same RTX PRO 6000. After Ever's visual-export fix, llm_inference was re-run on that box for robot_153.jpg and describe_anything.png, and the captions matched the images. Ever confirmed normal outputs on Thor. That retest did not name the tag as v0.11.0. This is now in the PR testing section.

The non-blocking notes are in f19c188, and I replied on those threads: one EDGELLM_ROOT default ($HOME/TensorRT-Edge-LLM), --viewPoint concat_view is accepted by the v0.11.0 binary, and policy engines are under ENGINE_DIR/policy.