Repository navigation
Conversation
Serving overlay for nvidia/Cosmos3-Nano and Cosmos3-Super on NVIDIA's public cosmos-framework, replacing the retired private EA fork (deepinfra/cosmos3): - worker.py: ported from the EA fork (import renames only); dynamo registration, schemas, gloo control-plane fix, torchrun relaunch and standalone mode unchanged. - Dockerfile: overlay build — clones cosmos-framework at a pinned ref, applies patches/, mirrors upstream's build, bakes worker + HF prefetch (incl. gated Cosmos-Guardrail1 for default-on guardrails). - patches/0001: encoder -threads cap (EA PR #19 re-applied). The 16 EA-era weight/code mismatch shims are obsolete upstream (native diffusers loading, AVAE released, parallelism-config bug fixed, guardrails default on). See examples/cosmos3/README.md for provenance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ARG values are recorded in image layer history, so a token passed that way is readable by anyone who can pull from the registry. The secret mount is available only during the RUN and never persists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- patches/0002: video guardrail runner drops the RetinaFace face-blur postprocessor, which unconditionally pixelates every detected face in every output frame (breaks the product; it is a privacy/anti-deepfake measure, not a content-safety control). The content-safety value stays: text-prompt guardrail (Blocklist + Qwen3Guard) is unchanged, and upstream already disables VideoContentSafetyFilter. Removal explicitly authorized by Johan (product owner) 2026-07-08. Flagged as potentially relevant to likeness/NCII/deepfake obligations; to be confirmed with legal as a follow-up (not a rollout blocker per Johan). - patches/0001 (encode-params): -threads cap plus explicit BT.709 signaling (colorspace+primaries+trc + limited range); conversion target and tag set together. Verified to round-trip identically to untagged on a compliant decoder — correctness hygiene, not a fix for a confirmed visible defect. - Dockerfile: apply patches + copy worker LAST (after the ~11GB gated prefetch), relying on the editable install, so patch iterations rebuild in seconds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In-image A/B on a color test card, both decoded as BT.709 by a compliant decoder: fw-v1 untagged (current): red 253,0,0 green 0,255,0 skin 239,193,165 (~exact) bt709/tv tagged (patch): red 255,23,0 green 0,215,0 skin 243,196,163 (err up to 40) Setting -colorspace/-color_range as imageio output flags TAGS the stream but does not correctly drive the RGB->YUV conversion, so tag and pixels disagree (the mislabel trap). The untagged output already round-trips cleanly on a compliant decoder, so there is no confirmed visible defect to fix here anyway. 0001 reverts to the thread-cap-only change. Proper color signaling (force the conversion via a scale filter matched to the tag) is deferred to its own task, gated on first confirming an actual browser-rendering problem exists. fw-v2 = face-blur removal (0002) + encoder thread cap (0001). No color change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…red imports) Formatting-only: black + isort reflow, and # noqa: E402 on the imports that must follow init_script() (the required init-before-heavy-imports pattern, same as cosmos_framework's own scripts). No behavior change — the built fw-v2 image (from the pre-format worker.py) is byte-behavior-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat: Cosmos3 videogen worker on public cosmos-framework (overlay)
Two post-merge follow-ups from the prod rollout: - Dockerfile: COPY worker.py to /opt/app/worker.py + WORKDIR /opt/app. MM2's dynamo deployment runs 'python3 worker.py' with workingDir=/opt/app, so the worker MUST be there — the prior /workspace location made every prod pod crash-loop (python3: can't open /opt/app/worker.py). This is the fix behind the fw-v3 rebuild that's now serving prod. - worker.py: replace 'except ValueError: pass' in Cosmos3VideoRequest.resolve() with a log.warning (GitHub Advanced Security / CodeQL empty-except finding). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: cosmos3 worker.py /opt/app path + CodeQL except:pass (follow-up to #14)
Set guardrails=False on the OmniSetupOverrides so the Cosmos worker skips the bundled text (Blocklist + Qwen3Guard) and video guardrail runners entirely. The Qwen3Guard text guard is unfit to gate customer prompts: - one coarse "Sexual Content or Sexual Acts" category, no CSAM/minor distinction, so it cannot express "block illegal only, allow legal"; - demonstrable demographic bias -- "an african american woman being picked up by a robot" is blocked as sexual while the same prompt with "white" returns normally; - the prompt check runs after generation (inference.py Save-outputs loop), so every rejected prompt still burns a full ~60s clip and surfaces as an opaque 500 "Failed to fold videos stream". Content filtering will move to a platform-level video-gen guardrail applied uniformly across all models, rather than a per-model guard here. Covers both Cosmos3-Nano and Cosmos3-Super (same worker). NVIDIA's own latency/throughput inference configs already run guardrails: False. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cosmos3: disable the bundled content guardrail
Guardrails are disabled in worker.py (guardrails=False, #21), so the nvidia/Cosmos-Guardrail1 snapshot (the only gated repo, ~10.8GB) is never loaded at runtime. Remove its prefetch, plus the now-unneeded BuildKit hf_token secret mount / HF_TOKEN export — the remaining prefetched repos (Qwen3 tokenizers, Wan VAEs) are all public (verified gated=False), so the build needs NO HF token. Effect: no token/gate hassle at build time, ~11GB smaller image, faster registry push/pull and pod cold-start. Runtime inference is unchanged (the weights were already unused post-#21). Re-add the download + secret if guardrails are ever turned back on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…efetch chore: drop gated Cosmos-Guardrail1 prefetch from cosmos3 worker image
… at boot) The framework's checkpoint_db shells out to `uvx hf@1.16.4 download ...` at model-init time, but the image baked/prefetched with hf@1.13.0. That version mismatch means uvx fetches hf 1.16.4 from PyPI on every pod boot (a runtime network dependency), and the prefetched HF cache (1.13.0 layout) isn't hit by the 1.16.4 runtime -> weights re-download from HF too. Align all hf invocations to 1.16.4 so the baked tool is reused and cache layout matches runtime. This is the tool half of removing the per-boot network dependency; the weights half (auxiliary encoder/VAE/sound-tokenizer served from deep-storage via a mounted HF cache + HF_HUB_OFFLINE=1) is a per-deployment change, not baked here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
chore: pin baked hf CLI to 1.16.4 to match framework (no PyPI fetch at boot)
Add COSMOS3_NUM_STEPS env (set via model-config extra_env) as the default denoising-step count, plus a per-request nvext.num_steps override. Lets us tune step count without an image rebuild. Falls back to the framework's per-modality default (35) when unset. Motivation: fewer steps is the one in-house, no-post-training cost lever for Cosmos3 video (sampling loop is ~94% of latency). Step-sweep on a reserved B200 (720p/5s Nano) confirmed clean latency scaling: 35=64s, 28=53s, 20=40s, 16=34s, 12=27s, 8=21s. 28 matches fal's default and looked visually identical to 35. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address adversarial review of the num_steps change: - Apply the default step count to VIDEO only (num_frames > 1). text2image defaults to 50 steps; the previous version silently forced it to the video value, an unvalidated quality cut on a live t2i path. Image modes now keep their framework default; explicit nvext.num_steps still overrides either. - Parse COSMOS3_NUM_STEPS defensively (non-int / <=0 -> warn + ignore) instead of int() at import, which would crash the worker fleet-wide on a bad value. - Keep all lines within the repo's 88-char limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat: config-driven sampling step count for cosmos3 worker
Signed-off-by: aidynamo-release-bot[bot] <292185679+aidynamo-release-bot[bot]@users.noreply.github.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
ai-dynamo#12440) Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
…dynamo#12427) Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com> Co-authored-by: Thomas Montfort <61255722+tmonty12@users.noreply.github.com>
…o#12480) Signed-off-by: Anna Tchernych <atchernych@nvidia.com> Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com> Co-authored-by: Anna Tchernych <atchernych@nvidia.com>
…namo#12509) Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
…o#12445) (ai-dynamo#12492) Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
…i-dynamo#12184) (ai-dynamo#12490) Signed-off-by: Schwinn Saereesitthipitak <galletas1712@users.noreply.github.com> Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
…-dynamo#12596) Signed-off-by: Connor Carpenter <connorc@nvidia.com>
…i-dynamo#12597) Signed-off-by: Connor Carpenter <connorc@nvidia.com>
…eases (cherry-pick ai-dynamo#13182) (ai-dynamo#13227) Signed-off-by: Dan Gil <dagil@nvidia.com>
…dynamo#13183) (ai-dynamo#13230) Signed-off-by: Dan Gil <dagil@nvidia.com>
…13016) (ai-dynamo#13225) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Co-authored-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>
…o#13256) Signed-off-by: Dan Gil <dagil@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
…#12950) Signed-off-by: Dan Gil <dagil@nvidia.com> Transient failures due to PyPI Infra
…lish (ai-dynamo#13307) Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>
… path (ai-dynamo#13314) Signed-off-by: Dan Gil <dagil@nvidia.com>
…[cherry-pick → release/1.4.0] (ai-dynamo#13432) Signed-off-by: PeaBrane <yanrpei@gmail.com> Co-authored-by: Yan Ru Pei <yanrpei@gmail.com>
…i-dynamo#13228) (ai-dynamo#13610) Signed-off-by: Julien Darve <jdarve@NVIDIA.com> Signed-off-by: Dan Gil <dagil@nvidia.com> Co-authored-by: JulienDarve <86800349+JulienDarve@users.noreply.github.com>
… frontend (ai-dynamo#12277) (ai-dynamo#13561) Signed-off-by: AineshSootha <aineshsootha@gmail.com> Signed-off-by: Dan Gil <dagil@nvidia.com> Co-authored-by: Ainesh Sootha <34103205+AineshSootha@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
…ai-dynamo#12994) (ai-dynamo#13563) Signed-off-by: Lavanya <lvijayakrish@nvidia.com> Signed-off-by: Dan Gil <dagil@nvidia.com> Co-authored-by: LAVANYA V <72541149+lavanyavijayk@users.noreply.github.com>
…ai-dynamo#12139, ai-dynamo#12142, ai-dynamo#12141) (ai-dynamo#13560) Signed-off-by: Jie Hao <jihao@nvidia.com> Signed-off-by: Dan Gil <dagil@nvidia.com> Co-authored-by: Jie Hao <jihao@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com>
containers=all wheels=all crates=all helm=all Signed-off-by: aidynamo-release-bot[bot] <292185679+aidynamo-release-bot[bot]@users.noreply.github.com>
containers=all wheels=all crates=all helm=all Signed-off-by: aidynamo-release-bot[bot] <292185679+aidynamo-release-bot[bot]@users.noreply.github.com>
…ing (ai-dynamo#13617) Signed-off-by: Dan Gil <dagil@nvidia.com>
…in across the go modules (ai-dynamo#13274) (ai-dynamo#13614) Signed-off-by: Dan Gil <dagil@nvidia.com>
…#13619) Signed-off-by: Dan Gil <dagil@nvidia.com>
…4.1) (ai-dynamo#13670) Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>
Syncs the fork with ai-dynamo/dynamo v1.4.1 (289 commits). The fork's own work is confined to examples/cosmos3/ and is untouched by this merge.
router_kv_hit_rate is a histogram of per-request overlap/isl, so sum/count is a mean of ratios. The engine's vllm:prefix_cache_hits/queries is a ratio of sums. On a skewed ISL distribution these diverge badly -- on DeepSeek-V4-Flash-0731 (44% of requests under 1800 tokens, mean 14211, tail past 130k) the router read 0.320 against the engine's 0.767 and looked half blind. Add router_kv_overlap_blocks_total / router_kv_isl_blocks_total so the router's predicted overlap can be compared with the engine directly. Measured on the same fleet right after deploy: block-weighted router 0.790 vs engine 0.767, i.e. the router's view was accurate all along. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pernekhan
temporarily deployed
to
external_collaborator
August 31, 2026 21:23 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
The metric help and the code comment pointed at vllm:prefix_cache_hits/queries. Every supported backend reports prefix-cache hits and queries as running totals, so the ratio-of-sums vs mean-of-ratios distinction these counters exist to close is not vLLM-specific. State it in terms of the backend engine instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pernekhan
had a problem deploying
to
external_collaborator
September 15, 2026 23:53 — with
GitHub Actions
Failure
GitHub Actions
Failure
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single commit of substance on top of
main, +36 lines across 3 files, no behavior change. Supersedes #34, which carried the same commit on top of 11 unrelated ones.The trap this closes
dynamo_component_router_kv_hit_rateis a histogram of per-requestoverlap_blocks / isl_blocks, sosum/countis a mean of ratios (request-weighted).Backend engines report prefix-cache hits and queries as running totals, so a hit rate derived from them is a ratio of sums (block-weighted). That holds for every supported backend, not just one of them.
The two aggregations are not comparable, and on a skewed ISL distribution they diverge badly. On a production model — 44% of requests under 1800 tokens (≤7 blocks), mean ISL 14,211, tail past 130k tokens (~508 blocks) — the router read 0.320 against the engine's 0.767 and looked half blind.
It isn't. That mis-comparison consumed most of a debugging session and nearly triggered an unnecessary indexer rewrite on a subsystem that was working correctly.
The fix
Add
router_kv_overlap_blocks_total/router_kv_isl_blocks_total, incremented at routing time fromselection.effective_overlap_blocksandisl_blocks. Dividing them yields a block-weighted router hit rate directly comparable to whatever the backend reports, with no per-backend arithmetic on the dashboard side.Verified live, immediately after deploy
Within 3% of the engine and slightly ahead, which is expected — some blocks are evicted between the routing decision and execution.
Independently corroborated: 10/10 paired requests sharing an 8k-token prefix showed 97.7% block-weighted reuse (range 96.0–98.5%), i.e. the router picked the prefix-holding worker every time.
Risk
None to serving. Two
IntCounters and their increments; no routing logic touched. The code comment states the aggregation difference explicitly so the next person doesn't walk into it.🤖 Generated with Claude Code