Skip to content

fix: admit SM120 and use FA4 with invalid-tile fix - #295

Draft
junghok-nv wants to merge 4 commits into
NVIDIA:mainfrom
junghok-nv:fix/sm120-gapped-forward-20261021
Draft

junghok-nv wants to merge 4 commits into
NVIDIA:mainfrom
junghok-nv:fix/sm120-gapped-forward-20261021

Conversation

@junghok-nv

@junghok-nv junghok-nv commented Oct 21, 2026 •

Copy link
Copy Markdown

Two bugs, two minimal fixes

  1. Framework rejects SM120 before execution. Replace one architecture condition in maskless_attention.py to admit exact (12, 0) alongside the existing SM90/SM10 targets.
  2. The pinned FA4 release contains the invalid-varlen-tile memory-access bug. Change the three FA4 dependency pins from 4.0.0b24 to 4.0.0b30, the first published release containing the already-merged FA4 fix #2763. Refresh only the necessary dependency closure in uv.lock.

The lock update needs CUTLASS DSL 4.6.2 and a compatible quack release because the old quack pin requires the old CUTLASS version. It does not upgrade Torch or the project's CUDA-runtime pins. No FA4 kernel is copied into this repository.

No runtime version checks, forward/backward restrictions, fallback policy, new helper modules, test files, documentation files, import reordering or unrelated refactoring are included.

Verification and limits

  • Exact-source checks: one architecture condition and three manifest version replacements only.
  • 13 CPU architecture-admission checks, Python syntax checks and git diff --check passed.
  • Targeted lock regeneration and uv lock --check passed with the project's required uv 0.11.29; unrelated package versions and the old non-FA4 CUTLASS/quack branches were retained.
  • All six non-installing dependency dry runs passed: cu128, cu130 and cu130-torch213, each on Linux x86_64 and aarch64.
  • No new GPU workload was run. Dependency resolution is not hardware qualification: SM120 inference and SM90/SM10 hardware regressions remain pending. Training/backward qualification is outside this patch's demonstrated scope. Keep this PR draft.

Signed-off-by: junghok <junghok@nvidia.com>
Signed-off-by: junghok <junghok@nvidia.com>
@junghok-nv junghok-nv changed the title Add guarded experimental SM120 gapped forward replay fix: allow guarded SM120 gapped forward replay Oct 21, 2026
Signed-off-by: junghok <junghok@nvidia.com>
@junghok-nv junghok-nv changed the title fix: allow guarded SM120 gapped forward replay fix: allow SM120 in gapped replay architecture check Oct 21, 2026
Signed-off-by: junghok <junghok@nvidia.com>
@junghok-nv junghok-nv changed the title fix: allow SM120 in gapped replay architecture check fix: admit SM120 and use FA4 with invalid-tile fix Oct 21, 2026

@pengcuo pengcuo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Draft, so just notes for when this is ready. I think this is the right shape of fix (see my review on #302, which solves the same sm120 refusal with a PyTorch fallback): FA4 already dispatches sm12x to its SM80 CuTe kernels for forward and backward, and Dao-AILab/flash-attention#2763 is in b30.

What I would want before un-drafting:

  • GPU evidence on sm120. The gate comment says the replay GPU tests only cover the sm90 and sm100 kernel paths. Please run cosmos_framework/model/generator/mot/multiview_maskless_exact_test.py and omni_mot_causal_model_test.py -k "replay or gapped" on an RTX PRO 6000 / RTX 50 with this branch (@hakuturu583 has the hardware and offered numbers in #302). The invariant that matters is that the SM80 kernel reads gap rows but never writes them.
  • Evidence that b30 + CUTLASS DSL 4.6.2 + quack 0.6.3 do not move sm90/sm100 outputs. CI needs a maintainer to approve the fork workflow, and ideally one Nano inference A/B against main on H100 or GB200 (bit-exact, or a stated diff).
  • Rebase. The PR is behind main (base a2a7409; main is 9a70e64).

Inline: the gate shape, the stale error text, and the pyproject comment that now contradicts the pin.

return "it needs NVIDIA CUDA, not ROCm"
major, minor = torch.cuda.get_device_capability(device)
if (major, minor) != (9, 0) and major != 10:
if (major, minor) not in ((9, 0), (12, 0)) and major != 10:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FA4 gates on arch // 10, so sm121 (DGX Spark / GB10) takes exactly the same kernels as sm120; admitting (12, 0) but refusing (12, 1) is an arbitrary split. Suggest major == 12 here (and major == 11 for Thor only if someone runs the tests there).

if (major, minor) not in ((9, 0), (12, 0)) and major != 10:
# Hopper and Blackwell use different FA4 kernels; the replay GPU tests cover
# the segment isolation and untouched gap rows required by both paths.
return f"it needs FlashAttention-4 on Hopper (sm90) or Blackwell (sm100/sm103), got sm{major}{minor}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message still lists only sm90/sm100/sm103 after the condition above started admitting sm120.

Suggested change
return f"it needs FlashAttention-4 on Hopper (sm90) or Blackwell (sm100/sm103), got sm{major}{minor}"
return f"it needs FlashAttention-4 on Hopper (sm90) or Blackwell (sm100/sm103/sm120), got sm{major}{minor}"

Comment thread pyproject.toml
@@ -231,7 +231,7 @@ cu128-train = [
# Match nvcr.io/nvidia/pytorch:25.11-py3: https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-25-11.html
cu130 = [
# Match imaginaire4's FA4 installation on both x86_64 and aarch64.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this bump the comment is no longer true unless imaginaire4's container moves to b30 as well. Please coordinate the pin with the i4 side (or drop the comment), since the two trees are kept in sync by the release flow.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants