Repository navigation

Rerun known transient failures once: HF/network errors and Blackwell CUDA crashes - #3059

Merged

kevalmorabia97 merged 9 commits into

mainfrom

kmorabia/skip-dsa-kv-cache-quant-blackwell

Oct 28, 2026

Merged

Rerun known transient failures once: HF/network errors and Blackwell CUDA crashes#3059
kevalmorabia97 merged 9 commits into
mainfrom
kmorabia/skip-dsa-kv-cache-quant-blackwell

Commits on Oct 27, 2026
- Skip the DSA KV-cache quantization test on Blackwell
  
  kevalmorabia97
  and
  claude
  committed
  390529a
- Run the flaky Blackwell megatron tests, tolerating only known CUDA crashes
  
  kevalmorabia97
  and
  claude
  committed
  b93f643
- Match only the CUDA error message, not worker deaths, in the Blackwell hook
  
  kevalmorabia97
  and
  claude
  committed
  ef28231
Commits on Oct 28, 2026
- Rerun known Blackwell CUDA crashes in gpu_megatron instead of xfailing them
  
  kevalmorabia97
  and
  claude
  committed
  20f7554
- Rerun Blackwell CUDA crashes only for tests that use a worker pool
  
  kevalmorabia97
  and
  claude
  committed
  bc8383b
- Rerun transient HF/network failures and Blackwell crashes with one mechanism
  
  kevalmorabia97
  and
  claude
  committed
  c7f3067
- Show the error behind each rerun in the pytest summary
  
  kevalmorabia97
  and
  claude
  committed
  1af7bf6
- Tighten the worker-abort rerun pattern and drop the unused plugin guard
  
  kevalmorabia97
  and
  claude
  committed
  ce71e35
- Add a TODO to remove the Blackwell crash rerun after the nemo container update
  
  kevalmorabia97
  and
  claude
  committed
  9617ed5