Skip to content

fix: make training importable and runnable on native Windows - #303

Draft
arif-ahmed-nv wants to merge 1 commit into
NVIDIA:mainfrom
arif-ahmed-nv:windows-compat
Draft

arif-ahmed-nv wants to merge 1 commit into
NVIDIA:mainfrom
arif-ahmed-nv:windows-compat

Conversation

@arif-ahmed-nv

Copy link
Copy Markdown

Summary

Stock cosmos-framework cannot be imported or started on Windows: several modules assume POSIX primitives at module scope or at trainer construction. This PR guards each assumption so that Linux behaviour is unchanged while a single-GPU native Windows run (x64 and Windows on ARM64 / RTX Spark) can import, train, evaluate and export.

All platform shims live in one new module, cosmos_framework/utils/win_compat.py, so the port is reviewable in one place. On Linux every helper there delegates to the stdlib primitive it replaces.

Fixes, one per POSIX assumption

Assumption Where Fix
os.sched_getaffinity / sched_setaffinity utils/distributed.py hasattr guard in _set_gpu_cpu_affinity; AttributeError added to the handler. On Windows the AttributeError is not an OSError, so it escaped the handler and aborted every entry point.
signal.SIGALRM / signal.alarm trainer/__init__.py Guarded; the step-timeout watchdog is disabled where alarm(2) does not exist instead of failing trainer init.
NCCL backend utils/distributed.py Fall back to gloo on win32 (is_nccl_available() is False). A process group is still mandatory: Trainer.__init__ runs an unconditional dist.barrier(). env:// defaults are set for the single-rank case, which has no torchrun launcher on native Windows.
gloo collectives on CUDA tensors utils/distributed.py, win_compat.py The Windows gloo build faults on CUDA tensors; collectives are staged through host memory.
DDP reducer on CUDA grads utils/distributed.py, trainer/__init__.py DDP wrapper skipped at world_size == 1 on win32 (access violation in backward otherwise). The trainer unwraps .module only when the wrapper was applied.
import fcntl (3 modules) lpips_cache.py, scripts/dataset_utils.py, utils/reasoner/distributed.py win_compat.flock (passthrough on Linux, msvcrt byte-range lock on Windows). In reasoner/distributed.py fcntl backs SIOCGIFADDR/SIOCGIFCONF interface discovery (multi-node only); imported defensively, helpers return "no interfaces" when absent.
os.symlink for the rank-0 job alias scripts/_train.py Falls back to a directory junction, then a pointer file (symlinks need Developer Mode or SeCreateSymbolicLinkPrivilege).
multistorageclient at module scope utils/easy_io/transient_retry.py Optional import (no Windows wheel or sdist; its xattr dependency needs POSIX xattrs). The backends package already guards it.
import triton at module scope model/generator/moe/native_kernels.py Optional import with a loud failure on kernel launch. Reached transitively by the config graph even for dense, non-MoE models; Triton has no official Windows wheel.

Also included, cross-platform: CheckpointDirHf drops a trivial --include * because the hf CLI parses a bare * as a positional filename, fetches only README.md, and still exits 0, which fails much later in the tokenizer.

Validation

Notes for reviewers

  • sys.platform == "win32" guards keep every behavioural change off the Linux path.
  • The pre-existing I001 import-order findings in touched files are unchanged by this PR.
  • The two remaining top-level Triton imports (mot/maskless_kernels.py, tokenizers/dc_ae) were not on the dense Nano path during validation and are left as is.

🤖 Generated with Claude Code

Stock cosmos-framework cannot be imported or started on Windows because
several modules assume POSIX primitives at module scope or at trainer
construction. This change guards each assumption so Linux behaviour is
byte-for-byte unchanged while a single-GPU Windows run can proceed.

Fixes, one per POSIX assumption:

- os.sched_getaffinity / sched_setaffinity: hasattr guard in
  _set_gpu_cpu_affinity, and AttributeError added to the handler. On Windows
  the AttributeError is not an OSError, so it escaped the existing handler
  and aborted every entry point before training started.
- signal.SIGALRM / signal.alarm: guarded; the step-timeout watchdog is
  disabled on platforms without alarm(2) instead of failing trainer init.
- NCCL backend: fall back to gloo on win32 (is_nccl_available() is False
  there) because Trainer.__init__ runs an unconditional dist.barrier(), so a
  process group is mandatory even at world_size 1. env:// rendezvous
  defaults are set for the single-rank case, which has no torchrun launcher
  on native Windows.
- gloo CUDA collectives: the Windows gloo build faults on CUDA tensors, so
  collectives are staged through host memory (win_compat shims).
- DDP wrapper: skipped at world_size == 1 on win32, where the reducer's
  allreduce on CUDA grads is an access violation in backward. The trainer
  unwraps `.module` only when the wrapper was applied.
- import fcntl (three modules): replaced by win_compat.flock, a passthrough
  on Linux and an msvcrt byte-range lock on Windows. In
  utils/reasoner/distributed.py fcntl backs SIOCGIFADDR/SIOCGIFCONF
  interface discovery (multi-node only); it is imported defensively and the
  helpers return "no interfaces" when it is absent.
- os.symlink for the rank-0 job alias: falls back to a directory junction,
  then a pointer file, since symlinks need Developer Mode or
  SeCreateSymbolicLinkPrivilege on Windows.
- multistorageclient: optional import in transient_retry.py (no Windows
  wheel or sdist; its xattr dependency needs POSIX xattrs). The backends
  package already guards it.
- triton: optional import in moe/native_kernels.py with a loud failure on
  kernel launch. The module is reached transitively by the config graph even
  for dense, non-MoE models, and Triton has no official Windows wheel.

Also included, cross-platform: CheckpointDirHf drops a trivial `--include *`
because the hf CLI parses a bare `*` as a positional filename, fetches only
README.md, and still exits 0.

Validated end to end (Cosmos3 Nano LoRA SFT, evaluate, export) on Windows 11
x64 and Windows 11 on ARM64 (RTX Spark) against the PR NVIDIA#164 base; this
rebase onto main (post NVIDIA#277) still needs the same run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Arif Ahmed <arahmed@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant