Repository navigation
fix: admit SM120 and use FA4 with invalid-tile fix - #295
junghok-nv wants to merge 4 commits into
Conversation
Signed-off-by: junghok <junghok@nvidia.com>
Signed-off-by: junghok <junghok@nvidia.com>
Signed-off-by: junghok <junghok@nvidia.com>
Signed-off-by: junghok <junghok@nvidia.com>
pengcuo
left a comment
There was a problem hiding this comment.
Draft, so just notes for when this is ready. I think this is the right shape of fix (see my review on #302, which solves the same sm120 refusal with a PyTorch fallback): FA4 already dispatches sm12x to its SM80 CuTe kernels for forward and backward, and Dao-AILab/flash-attention#2763 is in b30.
What I would want before un-drafting:
- GPU evidence on sm120. The gate comment says the replay GPU tests only cover the sm90 and sm100 kernel paths. Please run
cosmos_framework/model/generator/mot/multiview_maskless_exact_test.pyandomni_mot_causal_model_test.py -k "replay or gapped"on an RTX PRO 6000 / RTX 50 with this branch (@hakuturu583 has the hardware and offered numbers in #302). The invariant that matters is that the SM80 kernel reads gap rows but never writes them. - Evidence that b30 + CUTLASS DSL 4.6.2 + quack 0.6.3 do not move sm90/sm100 outputs. CI needs a maintainer to approve the fork workflow, and ideally one Nano inference A/B against
mainon H100 or GB200 (bit-exact, or a stated diff). - Rebase. The PR is behind
main(base a2a7409; main is 9a70e64).
Inline: the gate shape, the stale error text, and the pyproject comment that now contradicts the pin.
| return "it needs NVIDIA CUDA, not ROCm" | ||
| major, minor = torch.cuda.get_device_capability(device) | ||
| if (major, minor) != (9, 0) and major != 10: | ||
| if (major, minor) not in ((9, 0), (12, 0)) and major != 10: |
There was a problem hiding this comment.
FA4 gates on arch // 10, so sm121 (DGX Spark / GB10) takes exactly the same kernels as sm120; admitting (12, 0) but refusing (12, 1) is an arbitrary split. Suggest major == 12 here (and major == 11 for Thor only if someone runs the tests there).
| if (major, minor) not in ((9, 0), (12, 0)) and major != 10: | ||
| # Hopper and Blackwell use different FA4 kernels; the replay GPU tests cover | ||
| # the segment isolation and untouched gap rows required by both paths. | ||
| return f"it needs FlashAttention-4 on Hopper (sm90) or Blackwell (sm100/sm103), got sm{major}{minor}" |
There was a problem hiding this comment.
The message still lists only sm90/sm100/sm103 after the condition above started admitting sm120.
| return f"it needs FlashAttention-4 on Hopper (sm90) or Blackwell (sm100/sm103), got sm{major}{minor}" | |
| return f"it needs FlashAttention-4 on Hopper (sm90) or Blackwell (sm100/sm103/sm120), got sm{major}{minor}" |
| @@ -231,7 +231,7 @@ cu128-train = [ | |||
| # Match nvcr.io/nvidia/pytorch:25.11-py3: https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-25-11.html | |||
| cu130 = [ | |||
| # Match imaginaire4's FA4 installation on both x86_64 and aarch64. | |||
There was a problem hiding this comment.
After this bump the comment is no longer true unless imaginaire4's container moves to b30 as well. Please coordinate the pin with the i4 side (or drop the comment), since the two trees are kept in sync by the release flow.
Two bugs, two minimal fixes
maskless_attention.pyto admit exact(12, 0)alongside the existing SM90/SM10 targets.4.0.0b24to4.0.0b30, the first published release containing the already-merged FA4 fix #2763. Refresh only the necessary dependency closure inuv.lock.The lock update needs CUTLASS DSL 4.6.2 and a compatible quack release because the old quack pin requires the old CUTLASS version. It does not upgrade Torch or the project's CUDA-runtime pins. No FA4 kernel is copied into this repository.
No runtime version checks, forward/backward restrictions, fallback policy, new helper modules, test files, documentation files, import reordering or unrelated refactoring are included.
Verification and limits
git diff --checkpassed.uv lock --checkpassed with the project's required uv 0.11.29; unrelated package versions and the old non-FA4 CUTLASS/quack branches were retained.cu128,cu130andcu130-torch213, each on Linux x86_64 and aarch64.