Repository navigation
fix: make training importable and runnable on native Windows - #303
Draft
arif-ahmed-nv wants to merge 1 commit into
Draft
arif-ahmed-nv wants to merge 1 commit into
arif-ahmed-nv wants to merge 1 commit into
Conversation
Stock cosmos-framework cannot be imported or started on Windows because several modules assume POSIX primitives at module scope or at trainer construction. This change guards each assumption so Linux behaviour is byte-for-byte unchanged while a single-GPU Windows run can proceed. Fixes, one per POSIX assumption: - os.sched_getaffinity / sched_setaffinity: hasattr guard in _set_gpu_cpu_affinity, and AttributeError added to the handler. On Windows the AttributeError is not an OSError, so it escaped the existing handler and aborted every entry point before training started. - signal.SIGALRM / signal.alarm: guarded; the step-timeout watchdog is disabled on platforms without alarm(2) instead of failing trainer init. - NCCL backend: fall back to gloo on win32 (is_nccl_available() is False there) because Trainer.__init__ runs an unconditional dist.barrier(), so a process group is mandatory even at world_size 1. env:// rendezvous defaults are set for the single-rank case, which has no torchrun launcher on native Windows. - gloo CUDA collectives: the Windows gloo build faults on CUDA tensors, so collectives are staged through host memory (win_compat shims). - DDP wrapper: skipped at world_size == 1 on win32, where the reducer's allreduce on CUDA grads is an access violation in backward. The trainer unwraps `.module` only when the wrapper was applied. - import fcntl (three modules): replaced by win_compat.flock, a passthrough on Linux and an msvcrt byte-range lock on Windows. In utils/reasoner/distributed.py fcntl backs SIOCGIFADDR/SIOCGIFCONF interface discovery (multi-node only); it is imported defensively and the helpers return "no interfaces" when it is absent. - os.symlink for the rank-0 job alias: falls back to a directory junction, then a pointer file, since symlinks need Developer Mode or SeCreateSymbolicLinkPrivilege on Windows. - multistorageclient: optional import in transient_retry.py (no Windows wheel or sdist; its xattr dependency needs POSIX xattrs). The backends package already guards it. - triton: optional import in moe/native_kernels.py with a loud failure on kernel launch. The module is reached transitively by the config graph even for dense, non-MoE models, and Triton has no official Windows wheel. Also included, cross-platform: CheckpointDirHf drops a trivial `--include *` because the hf CLI parses a bare `*` as a positional filename, fetches only README.md, and still exits 0. Validated end to end (Cosmos3 Nano LoRA SFT, evaluate, export) on Windows 11 x64 and Windows 11 on ARM64 (RTX Spark) against the PR NVIDIA#164 base; this rebase onto main (post NVIDIA#277) still needs the same run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Arif Ahmed <arahmed@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stock cosmos-framework cannot be imported or started on Windows: several modules assume POSIX primitives at module scope or at trainer construction. This PR guards each assumption so that Linux behaviour is unchanged while a single-GPU native Windows run (x64 and Windows on ARM64 / RTX Spark) can import, train, evaluate and export.
All platform shims live in one new module,
cosmos_framework/utils/win_compat.py, so the port is reviewable in one place. On Linux every helper there delegates to the stdlib primitive it replaces.Fixes, one per POSIX assumption
os.sched_getaffinity/sched_setaffinityutils/distributed.pyhasattrguard in_set_gpu_cpu_affinity;AttributeErroradded to the handler. On Windows theAttributeErroris not anOSError, so it escaped the handler and aborted every entry point.signal.SIGALRM/signal.alarmtrainer/__init__.pyalarm(2)does not exist instead of failing trainer init.utils/distributed.pywin32(is_nccl_available()is False). A process group is still mandatory:Trainer.__init__runs an unconditionaldist.barrier().env://defaults are set for the single-rank case, which has no torchrun launcher on native Windows.utils/distributed.py,win_compat.pyutils/distributed.py,trainer/__init__.pyworld_size == 1onwin32(access violation in backward otherwise). The trainer unwraps.moduleonly when the wrapper was applied.import fcntl(3 modules)lpips_cache.py,scripts/dataset_utils.py,utils/reasoner/distributed.pywin_compat.flock(passthrough on Linux,msvcrtbyte-range lock on Windows). Inreasoner/distributed.pyfcntl backsSIOCGIFADDR/SIOCGIFCONFinterface discovery (multi-node only); imported defensively, helpers return "no interfaces" when absent.os.symlinkfor the rank-0 job aliasscripts/_train.pySeCreateSymbolicLinkPrivilege).multistorageclientat module scopeutils/easy_io/transient_retry.pyxattrdependency needs POSIX xattrs). The backends package already guards it.import tritonat module scopemodel/generator/moe/native_kernels.pyAlso included, cross-platform:
CheckpointDirHfdrops a trivial--include *because the hf CLI parses a bare*as a positional filename, fetches onlyREADME.md, and still exits 0, which fails much later in the tokenizer.Validation
main(after Add reproducible Cosmos3 Nano and Edge fine-tuning #277) compiles and passesruff format --check; the same end-to-end run onmainis still pending, which is why this is a draft.Notes for reviewers
sys.platform == "win32"guards keep every behavioural change off the Linux path.I001import-order findings in touched files are unchanged by this PR.mot/maskless_kernels.py,tokenizers/dc_ae) were not on the dense Nano path during validation and are left as is.🤖 Generated with Claude Code